{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "63766393",
   "metadata": {},
   "source": [
    "# Wide & Deep Recommendation for large scale data - Deep Learning Model Training\n",
    "\n",
    "This notebook demonstrates the distributed training for [Wide & Deep Learning](https://arxiv.org/abs/1606.07792) with the preprocessed features of the [Twitter Recsys Challenge 2021 dataset](http://www.recsyschallenge.com/2021/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "645c2461",
   "metadata": {},
   "source": [
    "<img src=\"figures/overview-recsys.png\" alt=\"overview-recsys\" width=\"750\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3824b94f",
   "metadata": {},
   "source": [
    "First of all, we import the necessary packages in BigDL for cluster initialization and distributed training. Also import TensorFlow for Wide & Deep model definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9c008424",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-23 10:55:39.569652: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2022-03-23 10:55:39.569679: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "from time import time\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "from bigdl.orca import init_orca_context, stop_orca_context, OrcaContext\n",
    "from bigdl.orca.learn.tf2.estimator import Estimator\n",
    "from bigdl.friesian.feature import FeatureTable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "998b3bb6",
   "metadata": {},
   "source": [
    "Initialize the environment on the YARN cluster. You simply need to prepare the Python environment on the driver node with [Anaconda](https://www.anaconda.com/products/individual) and BigDL will automatically distribute and prepare the environment for you across the cluster.\n",
    "Besides, you can specify the allocated resources for this application during the initialization, including the number of nodes, cores and the amount of memory to use, etc. BigDL provides detailed guidance to be easily deployed on [Hadoop/YARN](https://bigdl.readthedocs.io/en/latest/doc/UserGuide/hadoop.html) or [K8S](https://bigdl.readthedocs.io/en/latest/doc/UserGuide/k8s.html) clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d5e66b03",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initializing orca context\n",
      "Current pyspark location is : /root/anaconda3/envs/bigdl/lib/python3.7/site-packages/pyspark/__init__.py\n",
      "Initializing SparkContext for yarn-client mode\n",
      "Start to pack current python env\n",
      "Collecting packages...\n",
      "Packing environment at '/root/anaconda3/envs/bigdl' to '/tmp/tmpmq670r4z/python_env.tar.gz'\n",
      "[########################################] | 100% Completed | 13.4s\n",
      "Packing has been completed: /tmp/tmpmq670r4z/python_env.tar.gz\n",
      "pyspark_submit_args is: --master yarn --deploy-mode client --archives /tmp/tmpmq670r4z/python_env.tar.gz#python_env --driver-cores 4 --driver-memory 36g --num-executors 6 --executor-cores 28 --executor-memory 30g --driver-class-path /root/anaconda3/envs/bigdl/lib/python3.7/site-packages/bigdl/share/dllib/lib/bigdl-dllib-spark_2.4.6-2.0.0-jar-with-dependencies.jar:/root/anaconda3/envs/bigdl/lib/python3.7/site-packages/bigdl/share/orca/lib/bigdl-orca-spark_2.4.6-2.0.0-jar-with-dependencies.jar:/root/anaconda3/envs/bigdl/lib/python3.7/site-packages/bigdl/share/friesian/lib/bigdl-friesian-spark_2.4.6-2.0.0-jar-with-dependencies.jar pyspark-shell\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-03-23 10:55:59 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "2022-03-23 10:56:00 WARN  Client:66 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.\n",
      "2022-03-23 10:57:08,470 Thread-5 WARN The bufferSize is set to 4000 but bufferedIo is false: false\n",
      "2022-03-23 10:57:08,471 Thread-5 WARN The bufferSize is set to 4000 but bufferedIo is false: false\n",
      "2022-03-23 10:57:08,471 Thread-5 WARN The bufferSize is set to 4000 but bufferedIo is false: false\n",
      "2022-03-23 10:57:08,472 Thread-5 WARN The bufferSize is set to 4000 but bufferedIo is false: false\n",
      "22-03-23 10:57:08 [Thread-5] INFO  Engine$:121 - Auto detect executor number and executor cores number\n",
      "22-03-23 10:57:08 [Thread-5] INFO  Engine$:123 - Executor number is 6 and executor cores number is 28\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "User settings:\n",
      "\n",
      "   KMP_AFFINITY=granularity=fine,compact,1,0\n",
      "   KMP_BLOCKTIME=0\n",
      "   KMP_SETTINGS=1\n",
      "   OMP_NUM_THREADS=1\n",
      "\n",
      "Effective settings:\n",
      "\n",
      "   KMP_ABORT_DELAY=0\n",
      "   KMP_ADAPTIVE_LOCK_PROPS='1,1024'\n",
      "   KMP_ALIGN_ALLOC=64\n",
      "   KMP_ALL_THREADPRIVATE=224\n",
      "   KMP_ATOMIC_MODE=2\n",
      "   KMP_BLOCKTIME=0\n",
      "   KMP_CPUINFO_FILE: value is not defined\n",
      "   KMP_DETERMINISTIC_REDUCTION=false\n",
      "   KMP_DEVICE_THREAD_LIMIT=2147483647\n",
      "   KMP_DISP_HAND_THREAD=false\n",
      "   KMP_DISP_NUM_BUFFERS=7\n",
      "   KMP_DUPLICATE_LIB_OK=false\n",
      "   KMP_FORCE_REDUCTION: value is not defined\n",
      "   KMP_FOREIGN_THREADS_THREADPRIVATE=true\n",
      "   KMP_FORKJOIN_BARRIER='2,2'\n",
      "   KMP_FORKJOIN_BARRIER_PATTERN='hyper,hyper'\n",
      "   KMP_FORKJOIN_FRAMES=true\n",
      "   KMP_FORKJOIN_FRAMES_MODE=3\n",
      "   KMP_GTID_MODE=3\n",
      "   KMP_HANDLE_SIGNALS=false\n",
      "   KMP_HOT_TEAMS_MAX_LEVEL=1\n",
      "   KMP_HOT_TEAMS_MODE=0\n",
      "   KMP_INIT_AT_FORK=true\n",
      "   KMP_ITT_PREPARE_DELAY=0\n",
      "   KMP_LIBRARY=throughput\n",
      "   KMP_LOCK_KIND=queuing\n",
      "   KMP_MALLOC_POOL_INCR=1M\n",
      "   KMP_MWAIT_HINTS=0\n",
      "   KMP_NUM_LOCKS_IN_BLOCK=1\n",
      "   KMP_PLAIN_BARRIER='2,2'\n",
      "   KMP_PLAIN_BARRIER_PATTERN='hyper,hyper'\n",
      "   KMP_REDUCTION_BARRIER='1,1'\n",
      "   KMP_REDUCTION_BARRIER_PATTERN='hyper,hyper'\n",
      "   KMP_SCHEDULE='static,balanced;guided,iterative'\n",
      "   KMP_SETTINGS=true\n",
      "   KMP_SPIN_BACKOFF_PARAMS='4096,100'\n",
      "   KMP_STACKOFFSET=64\n",
      "   KMP_STACKPAD=0\n",
      "   KMP_STACKSIZE=8M\n",
      "   KMP_STORAGE_MAP=false\n",
      "   KMP_TASKING=2\n",
      "   KMP_TASKLOOP_MIN_TASKS=0\n",
      "   KMP_TASK_STEALING_CONSTRAINT=1\n",
      "   KMP_TEAMS_THREAD_LIMIT=56\n",
      "   KMP_TOPOLOGY_METHOD=all\n",
      "   KMP_USER_LEVEL_MWAIT=false\n",
      "   KMP_USE_YIELD=1\n",
      "   KMP_VERSION=false\n",
      "   KMP_WARNINGS=true\n",
      "   OMP_AFFINITY_FORMAT='OMP: pid %P tid %i thread %n bound to OS proc set {%A}'\n",
      "   OMP_ALLOCATOR=omp_default_mem_alloc\n",
      "   OMP_CANCELLATION=false\n",
      "   OMP_DEBUG=disabled\n",
      "   OMP_DEFAULT_DEVICE=0\n",
      "   OMP_DISPLAY_AFFINITY=false\n",
      "   OMP_DISPLAY_ENV=false\n",
      "   OMP_DYNAMIC=false\n",
      "   OMP_MAX_ACTIVE_LEVELS=2147483647\n",
      "   OMP_MAX_TASK_PRIORITY=0\n",
      "   OMP_NESTED=false\n",
      "   OMP_NUM_THREADS='1'\n",
      "   OMP_PLACES: value is not defined\n",
      "   OMP_PROC_BIND='intel'\n",
      "   OMP_SCHEDULE='static'\n",
      "   OMP_STACKSIZE=8M\n",
      "   OMP_TARGET_OFFLOAD=DEFAULT\n",
      "   OMP_THREAD_LIMIT=2147483647\n",
      "   OMP_TOOL=enabled\n",
      "   OMP_TOOL_LIBRARIES: value is not defined\n",
      "   OMP_WAIT_POLICY=PASSIVE\n",
      "   KMP_AFFINITY='noverbose,warnings,respect,granularity=fine,compact,1,0'\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22-03-23 10:57:08 [Thread-5] INFO  ThreadPool$:95 - Set mkl threads to 1 on thread 28\n",
      "2022-03-23 10:57:08 WARN  SparkContext:66 - Using an existing SparkContext; some configuration may not take effect.\n",
      "22-03-23 10:57:08 [Thread-5] INFO  Engine$:446 - Find existing spark context. Checking the spark conf...\n",
      "cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.Sample\n",
      "BigDLBasePickler registering: bigdl.dllib.utils.common  Sample\n",
      "cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.EvaluatedResult\n",
      "BigDLBasePickler registering: bigdl.dllib.utils.common  EvaluatedResult\n",
      "cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.JTensor\n",
      "BigDLBasePickler registering: bigdl.dllib.utils.common  JTensor\n",
      "cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.JActivity\n",
      "BigDLBasePickler registering: bigdl.dllib.utils.common  JActivity\n"
     ]
    }
   ],
   "source": [
    "# To display terminal's stdout and stderr in the Jupyter notebook.\n",
    "OrcaContext.log_output = True\n",
    "\n",
    "cluster_mode = \"yarn\"\n",
    "\n",
    "executor_cores = 28\n",
    "num_executor = 6\n",
    "executor_memory = \"30g\"\n",
    "driver_cores = 4\n",
    "driver_memory = \"36g\"\n",
    "conf = {\"spark.executor.memoryOverhead\": \"130g\",\n",
    "        \"spark.network.timeout\": \"10000000\",\n",
    "        \"spark.sql.broadcastTimeout\": \"7200\",\n",
    "        \"spark.sql.shuffle.partitions\": \"2000\",\n",
    "        \"spark.locality.wait\": \"0s\",\n",
    "        \"spark.sql.crossJoin.enabled\": \"true\",\n",
    "        \"spark.task.cpus\": \"1\",\n",
    "        \"spark.executor.heartbeatInterval\": \"200s\",\n",
    "        \"spark.driver.maxResultSize\": \"40G\",\n",
    "        \"spark.eventLog.enabled\": \"true\",\n",
    "        \"spark.app.name\": \"recsys-demo-train\"}\n",
    "\n",
    "if cluster_mode == \"local\":  # For local machine\n",
    "    sc = init_orca_context(cluster_mode=\"local\",\n",
    "                           cores=executor_cores, memory=executor_memory)\n",
    "elif cluster_mode == \"yarn\":  # For Hadoop/YARN cluster\n",
    "    sc = init_orca_context(cluster_mode=\"yarn\", cores=executor_cores,\n",
    "                           num_nodes=num_executor, memory=executor_memory,\n",
    "                           driver_cores=driver_cores, driver_memory=driver_memory,\n",
    "                           conf=conf, object_store_memory=\"80g\",\n",
    "                           env={\"KMP_BLOCKTIME\": \"1\",\n",
    "                                \"KMP_AFFINITY\": \"granularity=fine,compact,1,0\",\n",
    "                                \"OMP_NUM_THREADS\": \"28\"})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d073a018",
   "metadata": {},
   "source": [
    "Load the preprocessed train and validation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d6e25990",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "train_tbl = FeatureTable.read_parquet(\"/path/to/preprocessed/train/data\")\n",
    "valid_tbl = FeatureTable.read_parquet(\"/path/to/preprocessed/valid/data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "841d0f97",
   "metadata": {},
   "source": [
    "Define different features for Wide & Deep inputs:\n",
    "\n",
    "- wide_cols and cross_cols would be one hot encoded to be fed into the Linear layer.\n",
    "- embedding_cols would be converted to embedding vectors to be fed into the DNN.\n",
    "- indicator_cols would be one hot encoded to be fed into the DNN.\n",
    "- continuous_cols would be directly fed into the DNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8acf01e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "wide_cols = ['engaged_with_user_is_verified', 'enaging_user_is_verified']\n",
    "wide_dims = [1, 1]\n",
    "cross_cols = ['present_media_language']\n",
    "cross_dims = [600]\n",
    "\n",
    "embedding_cols = []\n",
    "embedding_dims = []\n",
    "\n",
    "cat_cols = ['present_media',\n",
    "            'tweet_type',\n",
    "            'language']\n",
    "cat_dims = [12, 2, 66]\n",
    "count_cols = ['engaged_with_user_follower_count',\n",
    "              'engaged_with_user_following_count',\n",
    "              'enaging_user_follower_count',\n",
    "              'enaging_user_following_count']\n",
    "count_dims = [7, 7, 7, 7]\n",
    "indicator_cols = cat_cols + count_cols\n",
    "indicator_dims = cat_dims + count_dims\n",
    "\n",
    "continuous_cols = ['len_hashtags',\n",
    "                   'len_domains',\n",
    "                   'len_links']\n",
    "\n",
    "column_info = { \"wide_base_cols\": wide_cols,\n",
    "                \"wide_base_dims\": wide_dims,\n",
    "                \"wide_cross_cols\": cross_cols,\n",
    "                \"wide_cross_dims\": cross_dims,\n",
    "                \"indicator_cols\": indicator_cols,\n",
    "                \"indicator_dims\": indicator_dims,\n",
    "                \"continuous_cols\": continuous_cols,\n",
    "                \"embed_cols\": [],\n",
    "                \"embed_in_dims\": [],\n",
    "                \"embed_out_dims\": [],\n",
    "                \"label\": \"label\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d3d30a6",
   "metadata": {},
   "source": [
    "Define the Wide & Deep model with TensorFlow Keras API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3293b38e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(column_info, hidden_units=[100, 50, 25]):\n",
    "    \"\"\"Build an estimator appropriate for the given model type.\"\"\"\n",
    "    wide_base_input_layers = []\n",
    "    wide_base_layers = []\n",
    "    for i in range(len(column_info[\"wide_base_cols\"])):\n",
    "        wide_base_input_layers.append(tf.keras.layers.Input(shape=[], dtype=\"int32\"))\n",
    "        wide_base_layers.append(tf.keras.backend.one_hot(wide_base_input_layers[i], column_info[\"wide_base_dims\"][i] + 1))\n",
    "\n",
    "    wide_cross_input_layers = []\n",
    "    wide_cross_layers = []\n",
    "    for i in range(len(column_info[\"wide_cross_cols\"])):\n",
    "        wide_cross_input_layers.append(tf.keras.layers.Input(shape=[], dtype=\"int32\"))\n",
    "        wide_cross_layers.append(tf.keras.backend.one_hot(wide_cross_input_layers[i], column_info[\"wide_cross_dims\"][i]))\n",
    "\n",
    "    indicator_input_layers = []\n",
    "    indicator_layers = []\n",
    "    for i in range(len(column_info[\"indicator_cols\"])):\n",
    "        indicator_input_layers.append(tf.keras.layers.Input(shape=[], dtype=\"int32\"))\n",
    "        indicator_layers.append(tf.keras.backend.one_hot(indicator_input_layers[i], column_info[\"indicator_dims\"][i] + 1))\n",
    "\n",
    "    embed_input_layers = []\n",
    "    embed_layers = []\n",
    "    for i in range(len(column_info[\"embed_in_dims\"])):\n",
    "        embed_input_layers.append(tf.keras.layers.Input(shape=[], dtype=\"int32\"))\n",
    "        iembed = tf.keras.layers.Embedding(column_info[\"embed_in_dims\"][i] + 1,\n",
    "                                           output_dim=column_info[\"embed_out_dims\"][i])(embed_input_layers[i])\n",
    "        flat_embed = tf.keras.layers.Flatten()(iembed)\n",
    "        embed_layers.append(flat_embed)\n",
    "\n",
    "    continuous_input_layers = []\n",
    "    continuous_layers = []\n",
    "    for i in range(len(column_info[\"continuous_cols\"])):\n",
    "        continuous_input_layers.append(tf.keras.layers.Input(shape=[]))\n",
    "        continuous_layers.append(tf.keras.layers.Reshape(target_shape=(1,))(continuous_input_layers[i]))\n",
    "\n",
    "    if len(wide_base_layers + wide_cross_layers) > 1:\n",
    "        wide_input = tf.keras.layers.concatenate(wide_base_layers + wide_cross_layers, axis=1)\n",
    "    else:\n",
    "        wide_input = (wide_base_layers + wide_cross_layers)[0]\n",
    "    wide_out = tf.keras.layers.Dense(1)(wide_input)\n",
    "    if len(indicator_layers + embed_layers + continuous_layers) > 1:\n",
    "        deep_concat = tf.keras.layers.concatenate(indicator_layers +\n",
    "                                                  embed_layers +\n",
    "                                                  continuous_layers, axis=1)\n",
    "    else:\n",
    "        deep_concat = (indicator_layers + embed_layers + continuous_layers)[0]\n",
    "    linear = deep_concat\n",
    "    for ilayer in range(0, len(hidden_units)):\n",
    "        linear_mid = tf.keras.layers.Dense(hidden_units[ilayer])(linear)\n",
    "        bn = tf.keras.layers.BatchNormalization()(linear_mid)\n",
    "        relu = tf.keras.layers.ReLU()(bn)\n",
    "        dropout = tf.keras.layers.Dropout(0.1)(relu)\n",
    "        linear = dropout\n",
    "    deep_out = tf.keras.layers.Dense(1)(linear)\n",
    "    added = tf.keras.layers.add([wide_out, deep_out])\n",
    "    out = tf.keras.layers.Activation(\"sigmoid\")(added)\n",
    "    model = tf.keras.models.Model(wide_base_input_layers +\n",
    "                                  wide_cross_input_layers +\n",
    "                                  indicator_input_layers +\n",
    "                                  embed_input_layers +\n",
    "                                  continuous_input_layers,\n",
    "                                  out)\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2009c87",
   "metadata": {},
   "source": [
    "The following figure shows the architecture of the Wide & Deep model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b737d69",
   "metadata": {},
   "source": [
    "<img src=\"figures/wnd.png\" alt=\"wnd\" width=\"750\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89034281",
   "metadata": {},
   "source": [
    "Set the hyperparameters for model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2fdc1edc",
   "metadata": {},
   "outputs": [],
   "source": [
    "config = {\n",
    "    \"lr\": 0.0001,\n",
    "    \"column_info\": column_info,\n",
    "    \"inter_op_parallelism\": 4,\n",
    "    \"intra_op_parallelism\": 24\n",
    "}\n",
    "batch_size = 25600"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8797ae41",
   "metadata": {},
   "source": [
    "Define the model_creator to create the Wide & Deep model on each node and compile it with optimizer, loss and metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8d51beb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def model_creator(config):\n",
    "    model = build_model(column_info=config[\"column_info\"],\n",
    "                        hidden_units=[1024, 1024])\n",
    "    optimizer = tf.keras.optimizers.Adam(config[\"lr\"])\n",
    "    model.compile(optimizer=optimizer,\n",
    "                  loss='binary_crossentropy',\n",
    "                  metrics=['binary_accuracy', 'binary_crossentropy', 'AUC', 'Precision', 'Recall'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "160bb526",
   "metadata": {},
   "source": [
    "The training architecture is shown in the following diagram. Our distributed training is implemented based on [RayOnSpark](https://medium.com/riselab/rayonspark-running-emerging-ai-applications-on-big-data-clusters-with-ray-and-analytics-zoo-923e0136ed6a) and [RaySGD](https://medium.com/distributed-computing-with-ray/faster-and-cheaper-pytorch-with-raysgd-a5a44d4fd220). On the underlying YARN cluster, the Spark Driver first launches multiple Spark executors across the cluster and the existing SparkContext creates a RayContext to launch Ray in the same cluster. The Ray processes stay alongside Spark executors and are managed by Spark executors. TensorFlow runners exist in Ray processes and they would take the local Spark in-memory data partitions for model training. TensorFlow runners communicate and synchronize with each other via [TensorFlow distribute MirroredStrategy](https://www.tensorflow.org/api_docs/python/tf/distribute/MirroredStrategy).\n",
    "\n",
    "The cluster setup and implementation details are transparent to users. Users simply need to create a TensorFlow Estimator in BigDL for the entire process."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19ca68d9",
   "metadata": {},
   "source": [
    "<img src=\"figures/e2etrain.png\" alt=\"e2etrain\" width=\"800\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b8d05eb",
   "metadata": {},
   "source": [
    "Create the TensorFlow Estimator in BigDL for distributed Wide & Deep training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "87a20abd",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Launching Ray on cluster with Spark barrier mode\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start to launch ray driver on local\n",
      "Executing command: ray start --address 172.168.0.107:38853 --redis-password 123456 --num-cpus 0 --object-store-memory 80000000000 --node-ip-address 172.168.0.101\n",
      "2022-03-23 10:58:43,531\tINFO scripts.py:747 -- Local node IP: 172.168.0.101\n",
      "2022-03-23 10:58:43,654\tSUCC scripts.py:755 -- --------------------\n",
      "2022-03-23 10:58:43,654\tSUCC scripts.py:756 -- Ray runtime started.\n",
      "2022-03-23 10:58:43,654\tSUCC scripts.py:757 -- --------------------\n",
      "2022-03-23 10:58:43,654\tINFO scripts.py:759 -- To terminate the Ray runtime, run\n",
      "2022-03-23 10:58:43,655\tINFO scripts.py:760 --   ray stop\n",
      "\n",
      "E0323 10:58:43.621758087  303499 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies\n",
      "E0323 10:58:43.629680754  303499 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies\n",
      "[2022-03-23 10:58:43,653 I 303499 303499] global_state_accessor.cc:360: This node has an IP address of 172.168.0.101, while we can not found the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.\n",
      "\n",
      "2022-03-23 10:58:43,531\tINFO scripts.py:747 -- Local node IP: 172.168.0.101\n",
      "2022-03-23 10:58:43,654\tSUCC scripts.py:755 -- --------------------\n",
      "2022-03-23 10:58:43,654\tSUCC scripts.py:756 -- Ray runtime started.\n",
      "2022-03-23 10:58:43,654\tSUCC scripts.py:757 -- --------------------\n",
      "2022-03-23 10:58:43,654\tINFO scripts.py:759 -- To terminate the Ray runtime, run\n",
      "2022-03-23 10:58:43,655\tINFO scripts.py:760 --   ray stop\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-23 10:58:44,960\tINFO worker.py:843 -- Connecting to existing Ray cluster at address: 172.168.0.107:38853\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'node_ip_address': '172.168.0.101', 'raylet_ip_address': '172.168.0.101', 'redis_address': '172.168.0.107:38853', 'object_store_address': '/tmp/ray/session_2022-03-23_10-58-37_224781_259004/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-03-23_10-58-37_224781_259004/sockets/raylet', 'webui_url': '172.168.0.107:8265', 'session_dir': '/tmp/ray/session_2022-03-23_10-58-37_224781_259004', 'metrics_export_port': 56881, 'node_id': '21217676fa1a3e4eee9e05be637197366ff94ddb13ed6ecc56433c9b'}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m 2022-03-23 10:58:46.251583: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: python_env/lib:python_env/lib/python3.7/lib-dynload::/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native\n",
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m 2022-03-23 10:58:46.251618: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m 2022-03-23 10:58:46.293903: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: python_env/lib:python_env/lib/python3.7/lib-dynload::/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m 2022-03-23 10:58:46.293939: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m 2022-03-23 10:58:46.288600: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: python_env/lib:python_env/lib/python3.7/lib-dynload::/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native\n",
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m 2022-03-23 10:58:46.288644: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m 2022-03-23 10:58:46.297479: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: python_env/lib:python_env/lib/python3.7/lib-dynload::/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m 2022-03-23 10:58:46.297511: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m 2022-03-23 10:58:46.290016: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: python_env/lib:python_env/lib/python3.7/lib-dynload::/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m 2022-03-23 10:58:46.290052: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m 2022-03-23 10:58:44.411926: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: python_env/lib:python_env/lib/python3.7/lib-dynload::/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m 2022-03-23 10:58:44.411962: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m WARNING:tensorflow:From /disk1/yarn/nm/usercache/root/appcache/application_1646615023158_0009/container_1646615023158_0009_01_000004/python_env/lib/python3.7/site-packages/bigdl/orca/learn/tf2/tf_runner.py:317: _CollectiveAllReduceStrategyExperimental.__init__ (from tensorflow.python.distribute.collective_all_reduce_strategy) is deprecated and will be removed in a future version.\n",
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m Instructions for updating:\n",
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m use distribute.MultiWorkerMirroredStrategy instead\n",
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m 2022-03-23 10:58:47.495565: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: python_env/lib:python_env/lib/python3.7/lib-dynload::/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native\n",
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m 2022-03-23 10:58:47.495639: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m 2022-03-23 10:58:47.495698: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (aep-007): /proc/driver/nvidia/version does not exist\n",
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m 2022-03-23 10:58:47.496959: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m WARNING:tensorflow:From /disk3/yarn/nm/usercache/root/appcache/application_1646615023158_0009/container_1646615023158_0009_01_000003/python_env/lib/python3.7/site-packages/bigdl/orca/learn/tf2/tf_runner.py:317: _CollectiveAllReduceStrategyExperimental.__init__ (from tensorflow.python.distribute.collective_all_reduce_strategy) is deprecated and will be removed in a future version.\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m Instructions for updating:\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m use distribute.MultiWorkerMirroredStrategy instead\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m 2022-03-23 10:58:47.505919: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: python_env/lib:python_env/lib/python3.7/lib-dynload::/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m 2022-03-23 10:58:47.505988: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m 2022-03-23 10:58:47.506030: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (aep-002): /proc/driver/nvidia/version does not exist\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m 2022-03-23 10:58:47.507324: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m WARNING:tensorflow:From /disk2/yarn/nm/usercache/root/appcache/application_1646615023158_0009/container_1646615023158_0009_01_000005/python_env/lib/python3.7/site-packages/bigdl/orca/learn/tf2/tf_runner.py:317: _CollectiveAllReduceStrategyExperimental.__init__ (from tensorflow.python.distribute.collective_all_reduce_strategy) is deprecated and will be removed in a future version.\n",
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m Instructions for updating:\n",
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m use distribute.MultiWorkerMirroredStrategy instead\n",
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m 2022-03-23 10:58:47.500270: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: python_env/lib:python_env/lib/python3.7/lib-dynload::/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native\n",
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m 2022-03-23 10:58:47.500360: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m 2022-03-23 10:58:47.500408: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (aep-003): /proc/driver/nvidia/version does not exist\n",
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m 2022-03-23 10:58:47.501713: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m WARNING:tensorflow:From /disk6/yarn/nm/usercache/root/appcache/application_1646615023158_0009/container_1646615023158_0009_01_000006/python_env/lib/python3.7/site-packages/bigdl/orca/learn/tf2/tf_runner.py:317: _CollectiveAllReduceStrategyExperimental.__init__ (from tensorflow.python.distribute.collective_all_reduce_strategy) is deprecated and will be removed in a future version.\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m Instructions for updating:\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m use distribute.MultiWorkerMirroredStrategy instead\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m 2022-03-23 10:58:47.499232: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: python_env/lib:python_env/lib/python3.7/lib-dynload::/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m 2022-03-23 10:58:47.499322: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m 2022-03-23 10:58:47.499374: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (aep-004): /proc/driver/nvidia/version does not exist\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m 2022-03-23 10:58:47.500623: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m 2022-03-23 10:58:47.537251: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job worker -> {0 -> 172.168.0.106:37715, 1 -> 172.168.0.103:33454, 2 -> 172.168.0.102:47470, 3 -> 172.168.0.107:51692, 4 -> 172.168.0.105:53809, 5 -> 172.168.0.104:45710}\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m 2022-03-23 10:58:47.537586: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:437] Started server with target: grpc://172.168.0.104:45710\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m WARNING:tensorflow:From /disk3/yarn/nm/usercache/root/appcache/application_1646615023158_0009/container_1646615023158_0009_01_000002/python_env/lib/python3.7/site-packages/bigdl/orca/learn/tf2/tf_runner.py:317: _CollectiveAllReduceStrategyExperimental.__init__ (from tensorflow.python.distribute.collective_all_reduce_strategy) is deprecated and will be removed in a future version.\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Instructions for updating:\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m use distribute.MultiWorkerMirroredStrategy instead\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m 2022-03-23 10:58:47.503409: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: python_env/lib:python_env/lib/python3.7/lib-dynload::/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m 2022-03-23 10:58:47.503489: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m 2022-03-23 10:58:47.503534: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (aep-006): /proc/driver/nvidia/version does not exist\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m 2022-03-23 10:58:47.504841: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m 2022-03-23 10:58:47.539125: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job worker -> {0 -> 172.168.0.106:37715, 1 -> 172.168.0.103:33454, 2 -> 172.168.0.102:47470, 3 -> 172.168.0.107:51692, 4 -> 172.168.0.105:53809, 5 -> 172.168.0.104:45710}\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m 2022-03-23 10:58:47.540202: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:437] Started server with target: grpc://172.168.0.106:37715\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m WARNING:tensorflow:From /disk7/yarn/nm/usercache/root/appcache/application_1646615023158_0009/container_1646615023158_0009_01_000007/python_env/lib/python3.7/site-packages/bigdl/orca/learn/tf2/tf_runner.py:317: _CollectiveAllReduceStrategyExperimental.__init__ (from tensorflow.python.distribute.collective_all_reduce_strategy) is deprecated and will be removed in a future version.\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m Instructions for updating:\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m use distribute.MultiWorkerMirroredStrategy instead\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m 2022-03-23 10:58:45.620762: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: python_env/lib:python_env/lib/python3.7/lib-dynload::/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/hadoop/lib/native\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m 2022-03-23 10:58:45.620833: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m 2022-03-23 10:58:45.620876: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (aep-005): /proc/driver/nvidia/version does not exist\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m 2022-03-23 10:58:45.622148: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m 2022-03-23 10:58:45.657672: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job worker -> {0 -> 172.168.0.106:37715, 1 -> 172.168.0.103:33454, 2 -> 172.168.0.102:47470, 3 -> 172.168.0.107:51692, 4 -> 172.168.0.105:53809, 5 -> 172.168.0.104:45710}\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m 2022-03-23 10:58:45.657991: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:437] Started server with target: grpc://172.168.0.105:53809\n",
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m 2022-03-23 10:58:47.531321: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job worker -> {0 -> 172.168.0.106:37715, 1 -> 172.168.0.103:33454, 2 -> 172.168.0.102:47470, 3 -> 172.168.0.107:51692, 4 -> 172.168.0.105:53809, 5 -> 172.168.0.104:45710}\n",
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m 2022-03-23 10:58:47.531643: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:437] Started server with target: grpc://172.168.0.107:51692\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m 2022-03-23 10:58:47.534996: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job worker -> {0 -> 172.168.0.106:37715, 1 -> 172.168.0.103:33454, 2 -> 172.168.0.102:47470, 3 -> 172.168.0.107:51692, 4 -> 172.168.0.105:53809, 5 -> 172.168.0.104:45710}\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m 2022-03-23 10:58:47.535338: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:437] Started server with target: grpc://172.168.0.102:47470\n",
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m 2022-03-23 10:58:47.549444: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job worker -> {0 -> 172.168.0.106:37715, 1 -> 172.168.0.103:33454, 2 -> 172.168.0.102:47470, 3 -> 172.168.0.107:51692, 4 -> 172.168.0.105:53809, 5 -> 172.168.0.104:45710}\n",
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m 2022-03-23 10:58:47.550417: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:437] Started server with target: grpc://172.168.0.103:33454\n"
     ]
    }
   ],
   "source": [
    "estimator = Estimator.from_keras(\n",
    "    model_creator=model_creator,\n",
    "    verbose=True,\n",
    "    config=config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cb90c82",
   "metadata": {},
   "source": [
    "We evaluate on the validation dataset every 50 iterations. We add an EarlyStopping callback so that when validation auc keeps dropping for three rounds, the training would be stopped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4c0c203b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-03-23 10:59:01 WARN  Utils:66 - Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of train records: 747694282\n",
      "Total number of val records: 14461760\n"
     ]
    }
   ],
   "source": [
    "train_count = train_tbl.size()\n",
    "print(\"Total number of train records: {}\".format(train_count))\n",
    "total_steps = math.ceil(train_count / batch_size)\n",
    "steps_per_epoch = 50\n",
    "epochs = math.ceil(total_steps / steps_per_epoch)   # To train the full dataset for an entire epoch\n",
    "val_count = valid_tbl.size()\n",
    "print(\"Total number of val records: {}\".format(val_count))\n",
    "val_steps = math.ceil(val_count / batch_size)\n",
    "\n",
    "callbacks = [EarlyStopping(monitor='val_auc', mode='max',verbose=1, patience=3)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f8d4e9e",
   "metadata": {},
   "source": [
    "Great! Let's launch the distributed training across the underlying YARN cluster! The training progress and the metrics at each step would be printed out during the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5b46a8fc",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Try to unpersist an uncached rdd\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 1/585\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m 2022-03-23 11:13:27.299909: W tensorflow/core/framework/dataset.cc:768] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 1/50 [..............................] - ETA: 4:20 - loss: 0.8724 - binary_accuracy: 0.4895 - binary_crossentropy: 0.8724 - auc: 0.4726 - precision: 0.4919 - recall: 0.7762\n",
      " 2/50 [>.............................] - ETA: 9s - loss: 0.8396 - binary_accuracy: 0.5014 - binary_crossentropy: 0.8396 - auc: 0.4978 - precision: 0.4987 - recall: 0.8051  \n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.8149 - binary_accuracy: 0.5174 - binary_crossentropy: 0.8149 - auc: 0.5247 - precision: 0.5102 - recall: 0.8144\n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.7979 - binary_accuracy: 0.5313 - binary_crossentropy: 0.7979 - auc: 0.5474 - precision: 0.5197 - recall: 0.8127\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.7878 - binary_accuracy: 0.5408 - binary_crossentropy: 0.7878 - auc: 0.5627 - precision: 0.5263 - recall: 0.8050\n",
      " 6/50 [==>...........................] - ETA: 8s - loss: 0.7790 - binary_accuracy: 0.5500 - binary_crossentropy: 0.7790 - auc: 0.5759 - precision: 0.5326 - recall: 0.7982\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.7726 - binary_accuracy: 0.5557 - binary_crossentropy: 0.7726 - auc: 0.5841 - precision: 0.5368 - recall: 0.7907\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.7660 - binary_accuracy: 0.5601 - binary_crossentropy: 0.7660 - auc: 0.5907 - precision: 0.5406 - recall: 0.7830\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.7602 - binary_accuracy: 0.5639 - binary_crossentropy: 0.7602 - auc: 0.5954 - precision: 0.5436 - recall: 0.7767\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.7549 - binary_accuracy: 0.5665 - binary_crossentropy: 0.7549 - auc: 0.5984 - precision: 0.5466 - recall: 0.7696\n",
      "11/50 [=====>........................] - ETA: 7s - loss: 0.7501 - binary_accuracy: 0.5687 - binary_crossentropy: 0.7501 - auc: 0.6009 - precision: 0.5487 - recall: 0.7635\n",
      "12/50 [======>.......................] - ETA: 7s - loss: 0.7455 - binary_accuracy: 0.5708 - binary_crossentropy: 0.7455 - auc: 0.6030 - precision: 0.5508 - recall: 0.7577\n",
      "13/50 [======>.......................] - ETA: 7s - loss: 0.7415 - binary_accuracy: 0.5722 - binary_crossentropy: 0.7415 - auc: 0.6047 - precision: 0.5521 - recall: 0.7517\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.7379 - binary_accuracy: 0.5736 - binary_crossentropy: 0.7379 - auc: 0.6058 - precision: 0.5539 - recall: 0.7455\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.7344 - binary_accuracy: 0.5749 - binary_crossentropy: 0.7344 - auc: 0.6069 - precision: 0.5554 - recall: 0.7397\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.7314 - binary_accuracy: 0.5759 - binary_crossentropy: 0.7314 - auc: 0.6078 - precision: 0.5566 - recall: 0.7337\n",
      "17/50 [=========>....................] - ETA: 6s - loss: 0.7284 - binary_accuracy: 0.5769 - binary_crossentropy: 0.7284 - auc: 0.6087 - precision: 0.5580 - recall: 0.7273\n",
      "18/50 [=========>....................] - ETA: 6s - loss: 0.7257 - binary_accuracy: 0.5779 - binary_crossentropy: 0.7257 - auc: 0.6097 - precision: 0.5592 - recall: 0.7214\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.7232 - binary_accuracy: 0.5790 - binary_crossentropy: 0.7232 - auc: 0.6107 - precision: 0.5607 - recall: 0.7157\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.7210 - binary_accuracy: 0.5800 - binary_crossentropy: 0.7210 - auc: 0.6115 - precision: 0.5622 - recall: 0.7100\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.7190 - binary_accuracy: 0.5808 - binary_crossentropy: 0.7190 - auc: 0.6121 - precision: 0.5635 - recall: 0.7045\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.7172 - binary_accuracy: 0.5815 - binary_crossentropy: 0.7172 - auc: 0.6128 - precision: 0.5647 - recall: 0.6990\n",
      "23/50 [============>.................] - ETA: 5s - loss: 0.7153 - binary_accuracy: 0.5822 - binary_crossentropy: 0.7153 - auc: 0.6135 - precision: 0.5660 - recall: 0.6936\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.7136 - binary_accuracy: 0.5831 - binary_crossentropy: 0.7136 - auc: 0.6144 - precision: 0.5673 - recall: 0.6888\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.7120 - binary_accuracy: 0.5837 - binary_crossentropy: 0.7120 - auc: 0.6151 - precision: 0.5683 - recall: 0.6839\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.7105 - binary_accuracy: 0.5843 - binary_crossentropy: 0.7105 - auc: 0.6157 - precision: 0.5695 - recall: 0.6790\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.7092 - binary_accuracy: 0.5849 - binary_crossentropy: 0.7092 - auc: 0.6162 - precision: 0.5706 - recall: 0.6745\n",
      "28/50 [===============>..............] - ETA: 4s - loss: 0.7080 - binary_accuracy: 0.5855 - binary_crossentropy: 0.7080 - auc: 0.6167 - precision: 0.5715 - recall: 0.6702\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.7068 - binary_accuracy: 0.5859 - binary_crossentropy: 0.7068 - auc: 0.6173 - precision: 0.5724 - recall: 0.6662\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.7056 - binary_accuracy: 0.5864 - binary_crossentropy: 0.7056 - auc: 0.6179 - precision: 0.5734 - recall: 0.6624\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.7046 - binary_accuracy: 0.5869 - binary_crossentropy: 0.7046 - auc: 0.6183 - precision: 0.5742 - recall: 0.6587\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.7035 - binary_accuracy: 0.5875 - binary_crossentropy: 0.7035 - auc: 0.6189 - precision: 0.5753 - recall: 0.6556\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.7024 - binary_accuracy: 0.5881 - binary_crossentropy: 0.7024 - auc: 0.6196 - precision: 0.5763 - recall: 0.6527\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.7012 - binary_accuracy: 0.5889 - binary_crossentropy: 0.7012 - auc: 0.6205 - precision: 0.5772 - recall: 0.6503\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.7003 - binary_accuracy: 0.5893 - binary_crossentropy: 0.7003 - auc: 0.6211 - precision: 0.5781 - recall: 0.6477\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6994 - binary_accuracy: 0.5897 - binary_crossentropy: 0.6994 - auc: 0.6217 - precision: 0.5786 - recall: 0.6456\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6985 - binary_accuracy: 0.5902 - binary_crossentropy: 0.6985 - auc: 0.6223 - precision: 0.5793 - recall: 0.6436\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6977 - binary_accuracy: 0.5905 - binary_crossentropy: 0.6977 - auc: 0.6227 - precision: 0.5800 - recall: 0.6419\n",
      "39/50 [======================>.......] - ETA: 2s - loss: 0.6970 - binary_accuracy: 0.5908 - binary_crossentropy: 0.6970 - auc: 0.6231 - precision: 0.5806 - recall: 0.6402\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6963 - binary_accuracy: 0.5912 - binary_crossentropy: 0.6963 - auc: 0.6236 - precision: 0.5812 - recall: 0.6389\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6956 - binary_accuracy: 0.5915 - binary_crossentropy: 0.6956 - auc: 0.6241 - precision: 0.5818 - recall: 0.6379\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6948 - binary_accuracy: 0.5920 - binary_crossentropy: 0.6948 - auc: 0.6247 - precision: 0.5825 - recall: 0.6371\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6942 - binary_accuracy: 0.5923 - binary_crossentropy: 0.6942 - auc: 0.6252 - precision: 0.5829 - recall: 0.6363\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6936 - binary_accuracy: 0.5927 - binary_crossentropy: 0.6936 - auc: 0.6257 - precision: 0.5833 - recall: 0.6357\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6929 - binary_accuracy: 0.5932 - binary_crossentropy: 0.6929 - auc: 0.6263 - precision: 0.5840 - recall: 0.6353\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6922 - binary_accuracy: 0.5935 - binary_crossentropy: 0.6922 - auc: 0.6269 - precision: 0.5844 - recall: 0.6349\n",
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6916 - binary_accuracy: 0.5939 - binary_crossentropy: 0.6916 - auc: 0.6274 - precision: 0.5849 - recall: 0.6345\n",
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6910 - binary_accuracy: 0.5943 - binary_crossentropy: 0.6910 - auc: 0.6279 - precision: 0.5854 - recall: 0.6343\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "49/50 [============================>.] - ETA: 0s - loss: 0.6904 - binary_accuracy: 0.5947 - binary_crossentropy: 0.6904 - auc: 0.6284 - precision: 0.5858 - recall: 0.6342\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6898 - binary_accuracy: 0.5950 - binary_crossentropy: 0.6898 - auc: 0.6289 - precision: 0.5862 - recall: 0.6340\n",
      "50/50 [==============================] - 45s 815ms/step - loss: 0.6898 - binary_accuracy: 0.5950 - binary_crossentropy: 0.6898 - auc: 0.6289 - precision: 0.5862 - recall: 0.6340 - val_loss: 0.6879 - val_binary_accuracy: 0.5504 - val_binary_crossentropy: 0.6879 - val_auc: 0.5803 - val_precision: 0.5845 - val_recall: 0.3878\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 2/585\n",
      " 1/50 [..............................] - ETA: 13s - loss: 0.6617 - binary_accuracy: 0.6115 - binary_crossentropy: 0.6617 - auc: 0.6546 - precision: 0.6065 - recall: 0.6300\n",
      " 2/50 [>.............................] - ETA: 10s - loss: 0.6624 - binary_accuracy: 0.6108 - binary_crossentropy: 0.6624 - auc: 0.6538 - precision: 0.6051 - recall: 0.6286\n",
      " 3/50 [>.............................] - ETA: 9s - loss: 0.6626 - binary_accuracy: 0.6113 - binary_crossentropy: 0.6626 - auc: 0.6534 - precision: 0.6050 - recall: 0.6295 \n",
      " 4/50 [=>............................] - ETA: 9s - loss: 0.6621 - binary_accuracy: 0.6115 - binary_crossentropy: 0.6621 - auc: 0.6541 - precision: 0.6052 - recall: 0.6305\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6625 - binary_accuracy: 0.6116 - binary_crossentropy: 0.6625 - auc: 0.6537 - precision: 0.6044 - recall: 0.6316\n",
      " 6/50 [==>...........................] - ETA: 8s - loss: 0.6623 - binary_accuracy: 0.6118 - binary_crossentropy: 0.6623 - auc: 0.6540 - precision: 0.6039 - recall: 0.6319\n",
      " 7/50 [===>..........................] - ETA: 8s - loss: 0.6619 - binary_accuracy: 0.6123 - binary_crossentropy: 0.6619 - auc: 0.6546 - precision: 0.6053 - recall: 0.6326\n",
      " 8/50 [===>..........................] - ETA: 8s - loss: 0.6622 - binary_accuracy: 0.6121 - binary_crossentropy: 0.6622 - auc: 0.6541 - precision: 0.6047 - recall: 0.6325\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6626 - binary_accuracy: 0.6117 - binary_crossentropy: 0.6626 - auc: 0.6538 - precision: 0.6038 - recall: 0.6328\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6624 - binary_accuracy: 0.6118 - binary_crossentropy: 0.6624 - auc: 0.6538 - precision: 0.6040 - recall: 0.6328\n",
      "11/50 [=====>........................] - ETA: 7s - loss: 0.6626 - binary_accuracy: 0.6116 - binary_crossentropy: 0.6626 - auc: 0.6534 - precision: 0.6040 - recall: 0.6323\n",
      "12/50 [======>.......................] - ETA: 6s - loss: 0.6621 - binary_accuracy: 0.6116 - binary_crossentropy: 0.6621 - auc: 0.6539 - precision: 0.6040 - recall: 0.6319\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6622 - binary_accuracy: 0.6115 - binary_crossentropy: 0.6622 - auc: 0.6538 - precision: 0.6045 - recall: 0.6312\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6620 - binary_accuracy: 0.6119 - binary_crossentropy: 0.6620 - auc: 0.6541 - precision: 0.6048 - recall: 0.6312\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6619 - binary_accuracy: 0.6122 - binary_crossentropy: 0.6619 - auc: 0.6543 - precision: 0.6053 - recall: 0.6310\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6616 - binary_accuracy: 0.6126 - binary_crossentropy: 0.6616 - auc: 0.6547 - precision: 0.6060 - recall: 0.6307\n",
      "17/50 [=========>....................] - ETA: 6s - loss: 0.6615 - binary_accuracy: 0.6128 - binary_crossentropy: 0.6615 - auc: 0.6549 - precision: 0.6064 - recall: 0.6305\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6616 - binary_accuracy: 0.6128 - binary_crossentropy: 0.6616 - auc: 0.6548 - precision: 0.6065 - recall: 0.6303\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6617 - binary_accuracy: 0.6128 - binary_crossentropy: 0.6617 - auc: 0.6548 - precision: 0.6065 - recall: 0.6298\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6615 - binary_accuracy: 0.6130 - binary_crossentropy: 0.6615 - auc: 0.6551 - precision: 0.6067 - recall: 0.6296\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6613 - binary_accuracy: 0.6134 - binary_crossentropy: 0.6613 - auc: 0.6554 - precision: 0.6070 - recall: 0.6295\n",
      "22/50 [============>.................] - ETA: 4s - loss: 0.6613 - binary_accuracy: 0.6136 - binary_crossentropy: 0.6613 - auc: 0.6555 - precision: 0.6072 - recall: 0.6292\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6611 - binary_accuracy: 0.6138 - binary_crossentropy: 0.6611 - auc: 0.6558 - precision: 0.6076 - recall: 0.6291\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6611 - binary_accuracy: 0.6137 - binary_crossentropy: 0.6611 - auc: 0.6557 - precision: 0.6077 - recall: 0.6282\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6611 - binary_accuracy: 0.6138 - binary_crossentropy: 0.6611 - auc: 0.6558 - precision: 0.6079 - recall: 0.6277\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6610 - binary_accuracy: 0.6140 - binary_crossentropy: 0.6610 - auc: 0.6559 - precision: 0.6084 - recall: 0.6274\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6608 - binary_accuracy: 0.6143 - binary_crossentropy: 0.6608 - auc: 0.6561 - precision: 0.6089 - recall: 0.6271\n",
      "28/50 [===============>..............] - ETA: 3s - loss: 0.6608 - binary_accuracy: 0.6143 - binary_crossentropy: 0.6608 - auc: 0.6561 - precision: 0.6089 - recall: 0.6269\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6607 - binary_accuracy: 0.6146 - binary_crossentropy: 0.6607 - auc: 0.6564 - precision: 0.6092 - recall: 0.6269\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6606 - binary_accuracy: 0.6146 - binary_crossentropy: 0.6606 - auc: 0.6565 - precision: 0.6094 - recall: 0.6268\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6606 - binary_accuracy: 0.6147 - binary_crossentropy: 0.6606 - auc: 0.6565 - precision: 0.6094 - recall: 0.6267\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6606 - binary_accuracy: 0.6146 - binary_crossentropy: 0.6606 - auc: 0.6564 - precision: 0.6094 - recall: 0.6263\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6606 - binary_accuracy: 0.6146 - binary_crossentropy: 0.6606 - auc: 0.6564 - precision: 0.6095 - recall: 0.6261\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6606 - binary_accuracy: 0.6146 - binary_crossentropy: 0.6606 - auc: 0.6565 - precision: 0.6094 - recall: 0.6260\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6606 - binary_accuracy: 0.6146 - binary_crossentropy: 0.6606 - auc: 0.6565 - precision: 0.6095 - recall: 0.6258\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6605 - binary_accuracy: 0.6147 - binary_crossentropy: 0.6605 - auc: 0.6566 - precision: 0.6096 - recall: 0.6257\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6604 - binary_accuracy: 0.6146 - binary_crossentropy: 0.6604 - auc: 0.6566 - precision: 0.6096 - recall: 0.6255\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6604 - binary_accuracy: 0.6147 - binary_crossentropy: 0.6604 - auc: 0.6567 - precision: 0.6097 - recall: 0.6254\n",
      "39/50 [======================>.......] - ETA: 1s - loss: 0.6603 - binary_accuracy: 0.6148 - binary_crossentropy: 0.6603 - auc: 0.6569 - precision: 0.6099 - recall: 0.6254\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6602 - binary_accuracy: 0.6150 - binary_crossentropy: 0.6602 - auc: 0.6571 - precision: 0.6102 - recall: 0.6254\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6602 - binary_accuracy: 0.6150 - binary_crossentropy: 0.6602 - auc: 0.6570 - precision: 0.6103 - recall: 0.6252\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6602 - binary_accuracy: 0.6150 - binary_crossentropy: 0.6602 - auc: 0.6570 - precision: 0.6102 - recall: 0.6252\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6601 - binary_accuracy: 0.6151 - binary_crossentropy: 0.6601 - auc: 0.6572 - precision: 0.6103 - recall: 0.6253\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6600 - binary_accuracy: 0.6152 - binary_crossentropy: 0.6600 - auc: 0.6573 - precision: 0.6105 - recall: 0.6254\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6599 - binary_accuracy: 0.6154 - binary_crossentropy: 0.6599 - auc: 0.6575 - precision: 0.6108 - recall: 0.6256\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6598 - binary_accuracy: 0.6154 - binary_crossentropy: 0.6598 - auc: 0.6576 - precision: 0.6108 - recall: 0.6257\n",
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6598 - binary_accuracy: 0.6154 - binary_crossentropy: 0.6598 - auc: 0.6576 - precision: 0.6106 - recall: 0.6257\n",
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6598 - binary_accuracy: 0.6155 - binary_crossentropy: 0.6598 - auc: 0.6577 - precision: 0.6107 - recall: 0.6257\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6598 - binary_accuracy: 0.6155 - binary_crossentropy: 0.6598 - auc: 0.6577 - precision: 0.6107 - recall: 0.6256\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6597 - binary_accuracy: 0.6156 - binary_crossentropy: 0.6597 - auc: 0.6579 - precision: 0.6108 - recall: 0.6256\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 793ms/step - loss: 0.6597 - binary_accuracy: 0.6156 - binary_crossentropy: 0.6597 - auc: 0.6579 - precision: 0.6108 - recall: 0.6256 - val_loss: 0.6846 - val_binary_accuracy: 0.5605 - val_binary_crossentropy: 0.6846 - val_auc: 0.6050 - val_precision: 0.6096 - val_recall: 0.3672\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 3/585\n",
      " 1/50 [..............................] - ETA: 11s - loss: 0.6547 - binary_accuracy: 0.6221 - binary_crossentropy: 0.6547 - auc: 0.6662 - precision: 0.6178 - recall: 0.6274\n",
      " 2/50 [>.............................] - ETA: 8s - loss: 0.6559 - binary_accuracy: 0.6194 - binary_crossentropy: 0.6559 - auc: 0.6633 - precision: 0.6135 - recall: 0.6232 \n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6553 - binary_accuracy: 0.6191 - binary_crossentropy: 0.6553 - auc: 0.6633 - precision: 0.6136 - recall: 0.6228\n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.6551 - binary_accuracy: 0.6189 - binary_crossentropy: 0.6551 - auc: 0.6632 - precision: 0.6127 - recall: 0.6242\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6556 - binary_accuracy: 0.6192 - binary_crossentropy: 0.6556 - auc: 0.6626 - precision: 0.6128 - recall: 0.6250\n",
      " 6/50 [==>...........................] - ETA: 7s - loss: 0.6557 - binary_accuracy: 0.6195 - binary_crossentropy: 0.6557 - auc: 0.6626 - precision: 0.6139 - recall: 0.6253\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6554 - binary_accuracy: 0.6200 - binary_crossentropy: 0.6554 - auc: 0.6631 - precision: 0.6151 - recall: 0.6258\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6547 - binary_accuracy: 0.6210 - binary_crossentropy: 0.6547 - auc: 0.6642 - precision: 0.6165 - recall: 0.6262\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6552 - binary_accuracy: 0.6208 - binary_crossentropy: 0.6552 - auc: 0.6638 - precision: 0.6159 - recall: 0.6263\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6552 - binary_accuracy: 0.6212 - binary_crossentropy: 0.6552 - auc: 0.6640 - precision: 0.6170 - recall: 0.6263\n",
      "11/50 [=====>........................] - ETA: 6s - loss: 0.6556 - binary_accuracy: 0.6208 - binary_crossentropy: 0.6556 - auc: 0.6637 - precision: 0.6163 - recall: 0.6259\n",
      "12/50 [======>.......................] - ETA: 6s - loss: 0.6557 - binary_accuracy: 0.6208 - binary_crossentropy: 0.6557 - auc: 0.6636 - precision: 0.6164 - recall: 0.6262\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6561 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6561 - auc: 0.6632 - precision: 0.6160 - recall: 0.6260\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6559 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6559 - auc: 0.6633 - precision: 0.6159 - recall: 0.6258\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6559 - binary_accuracy: 0.6206 - binary_crossentropy: 0.6559 - auc: 0.6634 - precision: 0.6158 - recall: 0.6261\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6558 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6558 - auc: 0.6635 - precision: 0.6160 - recall: 0.6259\n",
      "17/50 [=========>....................] - ETA: 5s - loss: 0.6557 - binary_accuracy: 0.6204 - binary_crossentropy: 0.6557 - auc: 0.6635 - precision: 0.6157 - recall: 0.6257\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6555 - binary_accuracy: 0.6206 - binary_crossentropy: 0.6555 - auc: 0.6637 - precision: 0.6161 - recall: 0.6259\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6556 - binary_accuracy: 0.6203 - binary_crossentropy: 0.6556 - auc: 0.6636 - precision: 0.6159 - recall: 0.6258\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6556 - binary_accuracy: 0.6203 - binary_crossentropy: 0.6556 - auc: 0.6636 - precision: 0.6161 - recall: 0.6256\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6554 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6554 - auc: 0.6638 - precision: 0.6164 - recall: 0.6259\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6556 - binary_accuracy: 0.6201 - binary_crossentropy: 0.6556 - auc: 0.6635 - precision: 0.6160 - recall: 0.6255\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6557 - binary_accuracy: 0.6201 - binary_crossentropy: 0.6557 - auc: 0.6634 - precision: 0.6164 - recall: 0.6256\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6555 - binary_accuracy: 0.6202 - binary_crossentropy: 0.6555 - auc: 0.6637 - precision: 0.6162 - recall: 0.6259\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6555 - binary_accuracy: 0.6203 - binary_crossentropy: 0.6555 - auc: 0.6638 - precision: 0.6161 - recall: 0.6260\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6554 - binary_accuracy: 0.6204 - binary_crossentropy: 0.6554 - auc: 0.6639 - precision: 0.6163 - recall: 0.6262\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6554 - binary_accuracy: 0.6206 - binary_crossentropy: 0.6554 - auc: 0.6639 - precision: 0.6163 - recall: 0.6266\n",
      "28/50 [===============>..............] - ETA: 3s - loss: 0.6554 - binary_accuracy: 0.6207 - binary_crossentropy: 0.6554 - auc: 0.6640 - precision: 0.6165 - recall: 0.6267\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6554 - binary_accuracy: 0.6206 - binary_crossentropy: 0.6554 - auc: 0.6639 - precision: 0.6164 - recall: 0.6267\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6554 - binary_accuracy: 0.6204 - binary_crossentropy: 0.6554 - auc: 0.6639 - precision: 0.6164 - recall: 0.6266\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6553 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6553 - auc: 0.6640 - precision: 0.6164 - recall: 0.6267\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6553 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6553 - auc: 0.6639 - precision: 0.6164 - recall: 0.6268\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6553 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6553 - auc: 0.6639 - precision: 0.6166 - recall: 0.6267\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6553 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6553 - auc: 0.6639 - precision: 0.6165 - recall: 0.6268\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6553 - binary_accuracy: 0.6204 - binary_crossentropy: 0.6553 - auc: 0.6639 - precision: 0.6165 - recall: 0.6268\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6552 - binary_accuracy: 0.6204 - binary_crossentropy: 0.6552 - auc: 0.6640 - precision: 0.6164 - recall: 0.6270\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6552 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6552 - auc: 0.6641 - precision: 0.6165 - recall: 0.6270\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6551 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6551 - auc: 0.6642 - precision: 0.6165 - recall: 0.6271\n",
      "39/50 [======================>.......] - ETA: 1s - loss: 0.6552 - binary_accuracy: 0.6204 - binary_crossentropy: 0.6552 - auc: 0.6641 - precision: 0.6164 - recall: 0.6272\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6551 - binary_accuracy: 0.6206 - binary_crossentropy: 0.6551 - auc: 0.6643 - precision: 0.6165 - recall: 0.6275\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6551 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6551 - auc: 0.6643 - precision: 0.6163 - recall: 0.6276\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6551 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6551 - auc: 0.6642 - precision: 0.6161 - recall: 0.6278\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6550 - binary_accuracy: 0.6206 - binary_crossentropy: 0.6550 - auc: 0.6643 - precision: 0.6163 - recall: 0.6278\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6550 - binary_accuracy: 0.6206 - binary_crossentropy: 0.6550 - auc: 0.6643 - precision: 0.6164 - recall: 0.6278\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6549 - binary_accuracy: 0.6206 - binary_crossentropy: 0.6549 - auc: 0.6644 - precision: 0.6165 - recall: 0.6278\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6549 - binary_accuracy: 0.6206 - binary_crossentropy: 0.6549 - auc: 0.6644 - precision: 0.6165 - recall: 0.6278\n",
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6549 - binary_accuracy: 0.6206 - binary_crossentropy: 0.6549 - auc: 0.6645 - precision: 0.6165 - recall: 0.6279\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6549 - binary_accuracy: 0.6205 - binary_crossentropy: 0.6549 - auc: 0.6644 - precision: 0.6164 - recall: 0.6278\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6549 - binary_accuracy: 0.6206 - binary_crossentropy: 0.6549 - auc: 0.6645 - precision: 0.6164 - recall: 0.6279\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6548 - binary_accuracy: 0.6207 - binary_crossentropy: 0.6548 - auc: 0.6646 - precision: 0.6166 - recall: 0.6281\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 797ms/step - loss: 0.6548 - binary_accuracy: 0.6207 - binary_crossentropy: 0.6548 - auc: 0.6646 - precision: 0.6166 - recall: 0.6281 - val_loss: 0.6790 - val_binary_accuracy: 0.5802 - val_binary_crossentropy: 0.6790 - val_auc: 0.6371 - val_precision: 0.6370 - val_recall: 0.3974\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 4/585\n",
      " 1/50 [..............................] - ETA: 9s - loss: 0.6535 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6535 - auc: 0.6668 - precision: 0.6217 - recall: 0.6360\n",
      " 2/50 [>.............................] - ETA: 10s - loss: 0.6525 - binary_accuracy: 0.6236 - binary_crossentropy: 0.6525 - auc: 0.6683 - precision: 0.6179 - recall: 0.6358\n",
      " 3/50 [>.............................] - ETA: 9s - loss: 0.6522 - binary_accuracy: 0.6229 - binary_crossentropy: 0.6522 - auc: 0.6685 - precision: 0.6175 - recall: 0.6343 \n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.6527 - binary_accuracy: 0.6219 - binary_crossentropy: 0.6527 - auc: 0.6677 - precision: 0.6184 - recall: 0.6316\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6535 - binary_accuracy: 0.6211 - binary_crossentropy: 0.6535 - auc: 0.6664 - precision: 0.6169 - recall: 0.6301\n",
      " 6/50 [==>...........................] - ETA: 8s - loss: 0.6539 - binary_accuracy: 0.6203 - binary_crossentropy: 0.6539 - auc: 0.6658 - precision: 0.6163 - recall: 0.6288\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6539 - binary_accuracy: 0.6207 - binary_crossentropy: 0.6539 - auc: 0.6657 - precision: 0.6169 - recall: 0.6290\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6538 - binary_accuracy: 0.6208 - binary_crossentropy: 0.6538 - auc: 0.6658 - precision: 0.6173 - recall: 0.6284\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6535 - binary_accuracy: 0.6212 - binary_crossentropy: 0.6535 - auc: 0.6663 - precision: 0.6176 - recall: 0.6288\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6535 - binary_accuracy: 0.6213 - binary_crossentropy: 0.6535 - auc: 0.6663 - precision: 0.6182 - recall: 0.6286\n",
      "11/50 [=====>........................] - ETA: 7s - loss: 0.6533 - binary_accuracy: 0.6216 - binary_crossentropy: 0.6533 - auc: 0.6666 - precision: 0.6184 - recall: 0.6290\n",
      "12/50 [======>.......................] - ETA: 7s - loss: 0.6532 - binary_accuracy: 0.6218 - binary_crossentropy: 0.6532 - auc: 0.6667 - precision: 0.6180 - recall: 0.6292\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6532 - binary_accuracy: 0.6220 - binary_crossentropy: 0.6532 - auc: 0.6668 - precision: 0.6183 - recall: 0.6292\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6530 - binary_accuracy: 0.6221 - binary_crossentropy: 0.6530 - auc: 0.6670 - precision: 0.6182 - recall: 0.6298\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6530 - binary_accuracy: 0.6223 - binary_crossentropy: 0.6530 - auc: 0.6671 - precision: 0.6184 - recall: 0.6302\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6529 - binary_accuracy: 0.6221 - binary_crossentropy: 0.6529 - auc: 0.6671 - precision: 0.6182 - recall: 0.6302\n",
      "17/50 [=========>....................] - ETA: 6s - loss: 0.6528 - binary_accuracy: 0.6224 - binary_crossentropy: 0.6528 - auc: 0.6673 - precision: 0.6183 - recall: 0.6306\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6527 - binary_accuracy: 0.6222 - binary_crossentropy: 0.6527 - auc: 0.6674 - precision: 0.6180 - recall: 0.6308\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6528 - binary_accuracy: 0.6223 - binary_crossentropy: 0.6528 - auc: 0.6673 - precision: 0.6180 - recall: 0.6307\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6529 - binary_accuracy: 0.6223 - binary_crossentropy: 0.6529 - auc: 0.6672 - precision: 0.6182 - recall: 0.6306\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6530 - binary_accuracy: 0.6223 - binary_crossentropy: 0.6530 - auc: 0.6671 - precision: 0.6183 - recall: 0.6306\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6530 - binary_accuracy: 0.6223 - binary_crossentropy: 0.6530 - auc: 0.6672 - precision: 0.6184 - recall: 0.6303\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6530 - binary_accuracy: 0.6222 - binary_crossentropy: 0.6530 - auc: 0.6671 - precision: 0.6184 - recall: 0.6299\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6530 - binary_accuracy: 0.6222 - binary_crossentropy: 0.6530 - auc: 0.6671 - precision: 0.6184 - recall: 0.6300\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6531 - binary_accuracy: 0.6222 - binary_crossentropy: 0.6531 - auc: 0.6669 - precision: 0.6184 - recall: 0.6299\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6532 - binary_accuracy: 0.6221 - binary_crossentropy: 0.6532 - auc: 0.6668 - precision: 0.6182 - recall: 0.6298\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6532 - binary_accuracy: 0.6219 - binary_crossentropy: 0.6532 - auc: 0.6668 - precision: 0.6181 - recall: 0.6296\n",
      "28/50 [===============>..............] - ETA: 4s - loss: 0.6532 - binary_accuracy: 0.6220 - binary_crossentropy: 0.6532 - auc: 0.6668 - precision: 0.6181 - recall: 0.6296\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6531 - binary_accuracy: 0.6221 - binary_crossentropy: 0.6531 - auc: 0.6669 - precision: 0.6182 - recall: 0.6299\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6530 - binary_accuracy: 0.6222 - binary_crossentropy: 0.6530 - auc: 0.6670 - precision: 0.6182 - recall: 0.6301\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6529 - binary_accuracy: 0.6223 - binary_crossentropy: 0.6529 - auc: 0.6672 - precision: 0.6182 - recall: 0.6302\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6529 - binary_accuracy: 0.6224 - binary_crossentropy: 0.6529 - auc: 0.6673 - precision: 0.6183 - recall: 0.6303\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6528 - binary_accuracy: 0.6224 - binary_crossentropy: 0.6528 - auc: 0.6674 - precision: 0.6182 - recall: 0.6305\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6527 - binary_accuracy: 0.6226 - binary_crossentropy: 0.6527 - auc: 0.6675 - precision: 0.6184 - recall: 0.6306\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6527 - binary_accuracy: 0.6226 - binary_crossentropy: 0.6527 - auc: 0.6675 - precision: 0.6184 - recall: 0.6305\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6527 - binary_accuracy: 0.6227 - binary_crossentropy: 0.6527 - auc: 0.6676 - precision: 0.6184 - recall: 0.6306\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6525 - binary_accuracy: 0.6229 - binary_crossentropy: 0.6525 - auc: 0.6678 - precision: 0.6186 - recall: 0.6307\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6525 - binary_accuracy: 0.6229 - binary_crossentropy: 0.6525 - auc: 0.6679 - precision: 0.6186 - recall: 0.6306\n",
      "39/50 [======================>.......] - ETA: 2s - loss: 0.6525 - binary_accuracy: 0.6230 - binary_crossentropy: 0.6525 - auc: 0.6679 - precision: 0.6187 - recall: 0.6307\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6524 - binary_accuracy: 0.6232 - binary_crossentropy: 0.6524 - auc: 0.6680 - precision: 0.6189 - recall: 0.6309\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6524 - binary_accuracy: 0.6233 - binary_crossentropy: 0.6524 - auc: 0.6680 - precision: 0.6192 - recall: 0.6308\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6523 - binary_accuracy: 0.6233 - binary_crossentropy: 0.6523 - auc: 0.6682 - precision: 0.6193 - recall: 0.6309\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6523 - binary_accuracy: 0.6234 - binary_crossentropy: 0.6523 - auc: 0.6683 - precision: 0.6193 - recall: 0.6308\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6522 - binary_accuracy: 0.6234 - binary_crossentropy: 0.6522 - auc: 0.6683 - precision: 0.6195 - recall: 0.6308\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6521 - binary_accuracy: 0.6235 - binary_crossentropy: 0.6521 - auc: 0.6685 - precision: 0.6195 - recall: 0.6309\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6521 - binary_accuracy: 0.6234 - binary_crossentropy: 0.6521 - auc: 0.6684 - precision: 0.6195 - recall: 0.6309\n",
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6521 - binary_accuracy: 0.6234 - binary_crossentropy: 0.6521 - auc: 0.6685 - precision: 0.6195 - recall: 0.6310\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6521 - binary_accuracy: 0.6233 - binary_crossentropy: 0.6521 - auc: 0.6684 - precision: 0.6194 - recall: 0.6310\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6521 - binary_accuracy: 0.6233 - binary_crossentropy: 0.6521 - auc: 0.6684 - precision: 0.6193 - recall: 0.6311\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6521 - binary_accuracy: 0.6233 - binary_crossentropy: 0.6521 - auc: 0.6684 - precision: 0.6192 - recall: 0.6312\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 792ms/step - loss: 0.6521 - binary_accuracy: 0.6233 - binary_crossentropy: 0.6521 - auc: 0.6684 - precision: 0.6192 - recall: 0.6312 - val_loss: 0.6744 - val_binary_accuracy: 0.5888 - val_binary_crossentropy: 0.6744 - val_auc: 0.6468 - val_precision: 0.6379 - val_recall: 0.4349\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 5/585\n",
      " 1/50 [..............................] - ETA: 9s - loss: 0.6526 - binary_accuracy: 0.6253 - binary_crossentropy: 0.6526 - auc: 0.6681 - precision: 0.6164 - recall: 0.6440\n",
      " 2/50 [>.............................] - ETA: 8s - loss: 0.6540 - binary_accuracy: 0.6211 - binary_crossentropy: 0.6540 - auc: 0.6656 - precision: 0.6155 - recall: 0.6381\n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6531 - binary_accuracy: 0.6224 - binary_crossentropy: 0.6531 - auc: 0.6673 - precision: 0.6166 - recall: 0.6384\n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.6527 - binary_accuracy: 0.6232 - binary_crossentropy: 0.6527 - auc: 0.6681 - precision: 0.6170 - recall: 0.6393\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6528 - binary_accuracy: 0.6228 - binary_crossentropy: 0.6528 - auc: 0.6680 - precision: 0.6162 - recall: 0.6374\n",
      " 6/50 [==>...........................] - ETA: 7s - loss: 0.6525 - binary_accuracy: 0.6226 - binary_crossentropy: 0.6525 - auc: 0.6683 - precision: 0.6160 - recall: 0.6356\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6523 - binary_accuracy: 0.6228 - binary_crossentropy: 0.6523 - auc: 0.6685 - precision: 0.6158 - recall: 0.6351\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6519 - binary_accuracy: 0.6232 - binary_crossentropy: 0.6519 - auc: 0.6689 - precision: 0.6167 - recall: 0.6342\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6520 - binary_accuracy: 0.6230 - binary_crossentropy: 0.6520 - auc: 0.6689 - precision: 0.6167 - recall: 0.6332\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6519 - binary_accuracy: 0.6230 - binary_crossentropy: 0.6519 - auc: 0.6690 - precision: 0.6171 - recall: 0.6326\n",
      "11/50 [=====>........................] - ETA: 7s - loss: 0.6515 - binary_accuracy: 0.6235 - binary_crossentropy: 0.6515 - auc: 0.6696 - precision: 0.6178 - recall: 0.6322\n",
      "12/50 [======>.......................] - ETA: 7s - loss: 0.6515 - binary_accuracy: 0.6237 - binary_crossentropy: 0.6515 - auc: 0.6696 - precision: 0.6186 - recall: 0.6319\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6514 - binary_accuracy: 0.6239 - binary_crossentropy: 0.6514 - auc: 0.6697 - precision: 0.6187 - recall: 0.6319\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6512 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6512 - auc: 0.6701 - precision: 0.6195 - recall: 0.6320\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6514 - binary_accuracy: 0.6244 - binary_crossentropy: 0.6514 - auc: 0.6699 - precision: 0.6198 - recall: 0.6318\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6512 - binary_accuracy: 0.6245 - binary_crossentropy: 0.6512 - auc: 0.6701 - precision: 0.6200 - recall: 0.6315\n",
      "17/50 [=========>....................] - ETA: 6s - loss: 0.6514 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6514 - auc: 0.6698 - precision: 0.6199 - recall: 0.6311\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6513 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6513 - auc: 0.6699 - precision: 0.6202 - recall: 0.6310\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6513 - binary_accuracy: 0.6245 - binary_crossentropy: 0.6513 - auc: 0.6699 - precision: 0.6203 - recall: 0.6312\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6515 - binary_accuracy: 0.6241 - binary_crossentropy: 0.6515 - auc: 0.6695 - precision: 0.6203 - recall: 0.6307\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6515 - binary_accuracy: 0.6240 - binary_crossentropy: 0.6515 - auc: 0.6694 - precision: 0.6201 - recall: 0.6307\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6515 - binary_accuracy: 0.6238 - binary_crossentropy: 0.6515 - auc: 0.6693 - precision: 0.6196 - recall: 0.6306\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6516 - binary_accuracy: 0.6237 - binary_crossentropy: 0.6516 - auc: 0.6692 - precision: 0.6193 - recall: 0.6308\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6515 - binary_accuracy: 0.6238 - binary_crossentropy: 0.6515 - auc: 0.6693 - precision: 0.6193 - recall: 0.6311\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6515 - binary_accuracy: 0.6238 - binary_crossentropy: 0.6515 - auc: 0.6693 - precision: 0.6193 - recall: 0.6312\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6516 - binary_accuracy: 0.6239 - binary_crossentropy: 0.6516 - auc: 0.6692 - precision: 0.6195 - recall: 0.6312\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6515 - binary_accuracy: 0.6239 - binary_crossentropy: 0.6515 - auc: 0.6694 - precision: 0.6195 - recall: 0.6314\n",
      "28/50 [===============>..............] - ETA: 4s - loss: 0.6515 - binary_accuracy: 0.6239 - binary_crossentropy: 0.6515 - auc: 0.6694 - precision: 0.6194 - recall: 0.6316\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6515 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6515 - auc: 0.6695 - precision: 0.6198 - recall: 0.6319\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6516 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6516 - auc: 0.6694 - precision: 0.6198 - recall: 0.6320\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6516 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6516 - auc: 0.6694 - precision: 0.6196 - recall: 0.6321\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6515 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6515 - auc: 0.6695 - precision: 0.6195 - recall: 0.6322\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6514 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6514 - auc: 0.6695 - precision: 0.6196 - recall: 0.6322\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6514 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6514 - auc: 0.6695 - precision: 0.6195 - recall: 0.6323\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6515 - binary_accuracy: 0.6241 - binary_crossentropy: 0.6515 - auc: 0.6693 - precision: 0.6193 - recall: 0.6322\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6514 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6514 - auc: 0.6695 - precision: 0.6194 - recall: 0.6322\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6515 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6515 - auc: 0.6694 - precision: 0.6196 - recall: 0.6322\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6514 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6514 - auc: 0.6695 - precision: 0.6197 - recall: 0.6322\n",
      "39/50 [======================>.......] - ETA: 2s - loss: 0.6514 - binary_accuracy: 0.6241 - binary_crossentropy: 0.6514 - auc: 0.6694 - precision: 0.6195 - recall: 0.6321\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6514 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6514 - auc: 0.6694 - precision: 0.6195 - recall: 0.6322\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6514 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6514 - auc: 0.6694 - precision: 0.6195 - recall: 0.6322\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6514 - binary_accuracy: 0.6241 - binary_crossentropy: 0.6514 - auc: 0.6694 - precision: 0.6193 - recall: 0.6322\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6514 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6514 - auc: 0.6694 - precision: 0.6196 - recall: 0.6322\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6513 - binary_accuracy: 0.6244 - binary_crossentropy: 0.6513 - auc: 0.6695 - precision: 0.6197 - recall: 0.6323\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6513 - binary_accuracy: 0.6244 - binary_crossentropy: 0.6513 - auc: 0.6695 - precision: 0.6197 - recall: 0.6322\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6513 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6513 - auc: 0.6694 - precision: 0.6196 - recall: 0.6322\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6513 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6513 - auc: 0.6695 - precision: 0.6197 - recall: 0.6323\n",
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6514 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6514 - auc: 0.6694 - precision: 0.6195 - recall: 0.6323\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6513 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6513 - auc: 0.6694 - precision: 0.6197 - recall: 0.6323\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6513 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6513 - auc: 0.6694 - precision: 0.6198 - recall: 0.6322\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 38s 781ms/step - loss: 0.6513 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6513 - auc: 0.6694 - precision: 0.6198 - recall: 0.6322 - val_loss: 0.6686 - val_binary_accuracy: 0.5998 - val_binary_crossentropy: 0.6686 - val_auc: 0.6607 - val_precision: 0.6594 - val_recall: 0.4336\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 6/585\n",
      " 1/50 [..............................] - ETA: 12s - loss: 0.6524 - binary_accuracy: 0.6213 - binary_crossentropy: 0.6524 - auc: 0.6658 - precision: 0.6171 - recall: 0.6241\n",
      " 2/50 [>.............................] - ETA: 10s - loss: 0.6519 - binary_accuracy: 0.6214 - binary_crossentropy: 0.6519 - auc: 0.6671 - precision: 0.6201 - recall: 0.6238\n",
      " 3/50 [>.............................] - ETA: 9s - loss: 0.6518 - binary_accuracy: 0.6217 - binary_crossentropy: 0.6518 - auc: 0.6679 - precision: 0.6177 - recall: 0.6259 \n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.6506 - binary_accuracy: 0.6233 - binary_crossentropy: 0.6506 - auc: 0.6700 - precision: 0.6190 - recall: 0.6292\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6508 - binary_accuracy: 0.6232 - binary_crossentropy: 0.6508 - auc: 0.6698 - precision: 0.6189 - recall: 0.6304\n",
      " 6/50 [==>...........................] - ETA: 8s - loss: 0.6515 - binary_accuracy: 0.6229 - binary_crossentropy: 0.6515 - auc: 0.6690 - precision: 0.6186 - recall: 0.6312\n",
      " 7/50 [===>..........................] - ETA: 8s - loss: 0.6515 - binary_accuracy: 0.6230 - binary_crossentropy: 0.6515 - auc: 0.6692 - precision: 0.6180 - recall: 0.6323\n",
      " 8/50 [===>..........................] - ETA: 8s - loss: 0.6515 - binary_accuracy: 0.6231 - binary_crossentropy: 0.6515 - auc: 0.6690 - precision: 0.6181 - recall: 0.6324\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6514 - binary_accuracy: 0.6234 - binary_crossentropy: 0.6514 - auc: 0.6691 - precision: 0.6186 - recall: 0.6330\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6510 - binary_accuracy: 0.6238 - binary_crossentropy: 0.6510 - auc: 0.6696 - precision: 0.6191 - recall: 0.6335\n",
      "11/50 [=====>........................] - ETA: 7s - loss: 0.6513 - binary_accuracy: 0.6237 - binary_crossentropy: 0.6513 - auc: 0.6692 - precision: 0.6185 - recall: 0.6333\n",
      "12/50 [======>.......................] - ETA: 7s - loss: 0.6511 - binary_accuracy: 0.6238 - binary_crossentropy: 0.6511 - auc: 0.6695 - precision: 0.6191 - recall: 0.6333\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6506 - binary_accuracy: 0.6244 - binary_crossentropy: 0.6506 - auc: 0.6702 - precision: 0.6196 - recall: 0.6339\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6506 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6506 - auc: 0.6702 - precision: 0.6195 - recall: 0.6340\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6505 - binary_accuracy: 0.6247 - binary_crossentropy: 0.6505 - auc: 0.6705 - precision: 0.6200 - recall: 0.6342\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6508 - binary_accuracy: 0.6245 - binary_crossentropy: 0.6508 - auc: 0.6702 - precision: 0.6197 - recall: 0.6340\n",
      "17/50 [=========>....................] - ETA: 6s - loss: 0.6509 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6509 - auc: 0.6701 - precision: 0.6193 - recall: 0.6338\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6510 - binary_accuracy: 0.6241 - binary_crossentropy: 0.6510 - auc: 0.6700 - precision: 0.6193 - recall: 0.6337\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6508 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6508 - auc: 0.6703 - precision: 0.6197 - recall: 0.6337\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6506 - binary_accuracy: 0.6244 - binary_crossentropy: 0.6506 - auc: 0.6704 - precision: 0.6198 - recall: 0.6337\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6508 - binary_accuracy: 0.6241 - binary_crossentropy: 0.6508 - auc: 0.6701 - precision: 0.6194 - recall: 0.6333\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6506 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6506 - auc: 0.6703 - precision: 0.6195 - recall: 0.6334\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6506 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6506 - auc: 0.6703 - precision: 0.6198 - recall: 0.6334\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6505 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6505 - auc: 0.6705 - precision: 0.6198 - recall: 0.6334\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6504 - binary_accuracy: 0.6244 - binary_crossentropy: 0.6504 - auc: 0.6706 - precision: 0.6199 - recall: 0.6335\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6503 - binary_accuracy: 0.6245 - binary_crossentropy: 0.6503 - auc: 0.6708 - precision: 0.6200 - recall: 0.6336\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6503 - binary_accuracy: 0.6247 - binary_crossentropy: 0.6503 - auc: 0.6709 - precision: 0.6201 - recall: 0.6338\n",
      "28/50 [===============>..............] - ETA: 4s - loss: 0.6503 - binary_accuracy: 0.6246 - binary_crossentropy: 0.6503 - auc: 0.6709 - precision: 0.6202 - recall: 0.6339\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6501 - binary_accuracy: 0.6248 - binary_crossentropy: 0.6501 - auc: 0.6711 - precision: 0.6204 - recall: 0.6342\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6502 - binary_accuracy: 0.6248 - binary_crossentropy: 0.6502 - auc: 0.6710 - precision: 0.6203 - recall: 0.6343\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6502 - binary_accuracy: 0.6247 - binary_crossentropy: 0.6502 - auc: 0.6709 - precision: 0.6202 - recall: 0.6343\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6502 - binary_accuracy: 0.6247 - binary_crossentropy: 0.6502 - auc: 0.6709 - precision: 0.6201 - recall: 0.6344\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6502 - binary_accuracy: 0.6248 - binary_crossentropy: 0.6502 - auc: 0.6710 - precision: 0.6202 - recall: 0.6346\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6502 - binary_accuracy: 0.6248 - binary_crossentropy: 0.6502 - auc: 0.6710 - precision: 0.6202 - recall: 0.6347\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6501 - binary_accuracy: 0.6249 - binary_crossentropy: 0.6501 - auc: 0.6710 - precision: 0.6202 - recall: 0.6348\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6501 - binary_accuracy: 0.6250 - binary_crossentropy: 0.6501 - auc: 0.6711 - precision: 0.6204 - recall: 0.6348\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6501 - binary_accuracy: 0.6251 - binary_crossentropy: 0.6501 - auc: 0.6712 - precision: 0.6205 - recall: 0.6348\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6503 - binary_accuracy: 0.6248 - binary_crossentropy: 0.6503 - auc: 0.6708 - precision: 0.6204 - recall: 0.6346\n",
      "39/50 [======================>.......] - ETA: 2s - loss: 0.6503 - binary_accuracy: 0.6249 - binary_crossentropy: 0.6503 - auc: 0.6709 - precision: 0.6204 - recall: 0.6348\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6502 - binary_accuracy: 0.6249 - binary_crossentropy: 0.6502 - auc: 0.6709 - precision: 0.6204 - recall: 0.6349\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6503 - binary_accuracy: 0.6249 - binary_crossentropy: 0.6503 - auc: 0.6708 - precision: 0.6203 - recall: 0.6350\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6503 - binary_accuracy: 0.6248 - binary_crossentropy: 0.6503 - auc: 0.6707 - precision: 0.6202 - recall: 0.6348\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6504 - binary_accuracy: 0.6246 - binary_crossentropy: 0.6504 - auc: 0.6706 - precision: 0.6199 - recall: 0.6348\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6504 - binary_accuracy: 0.6246 - binary_crossentropy: 0.6504 - auc: 0.6706 - precision: 0.6198 - recall: 0.6348\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6503 - binary_accuracy: 0.6247 - binary_crossentropy: 0.6503 - auc: 0.6706 - precision: 0.6200 - recall: 0.6347\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6504 - binary_accuracy: 0.6247 - binary_crossentropy: 0.6504 - auc: 0.6705 - precision: 0.6200 - recall: 0.6346\n",
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6504 - binary_accuracy: 0.6247 - binary_crossentropy: 0.6504 - auc: 0.6706 - precision: 0.6200 - recall: 0.6345\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6504 - binary_accuracy: 0.6247 - binary_crossentropy: 0.6504 - auc: 0.6706 - precision: 0.6200 - recall: 0.6345\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6504 - binary_accuracy: 0.6247 - binary_crossentropy: 0.6504 - auc: 0.6706 - precision: 0.6200 - recall: 0.6344\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6504 - binary_accuracy: 0.6246 - binary_crossentropy: 0.6504 - auc: 0.6706 - precision: 0.6199 - recall: 0.6343\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 41s 827ms/step - loss: 0.6504 - binary_accuracy: 0.6246 - binary_crossentropy: 0.6504 - auc: 0.6706 - precision: 0.6199 - recall: 0.6343 - val_loss: 0.6596 - val_binary_accuracy: 0.6189 - val_binary_crossentropy: 0.6596 - val_auc: 0.6748 - val_precision: 0.6550 - val_recall: 0.5225\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 7/585\n",
      " 1/50 [..............................] - ETA: 11s - loss: 0.6478 - binary_accuracy: 0.6271 - binary_crossentropy: 0.6478 - auc: 0.6737 - precision: 0.6190 - recall: 0.6316\n",
      " 2/50 [>.............................] - ETA: 8s - loss: 0.6489 - binary_accuracy: 0.6255 - binary_crossentropy: 0.6489 - auc: 0.6718 - precision: 0.6169 - recall: 0.6298 \n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6484 - binary_accuracy: 0.6262 - binary_crossentropy: 0.6484 - auc: 0.6728 - precision: 0.6198 - recall: 0.6300\n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.6481 - binary_accuracy: 0.6265 - binary_crossentropy: 0.6481 - auc: 0.6733 - precision: 0.6194 - recall: 0.6309\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6484 - binary_accuracy: 0.6262 - binary_crossentropy: 0.6484 - auc: 0.6730 - precision: 0.6194 - recall: 0.6300\n",
      " 6/50 [==>...........................] - ETA: 7s - loss: 0.6480 - binary_accuracy: 0.6266 - binary_crossentropy: 0.6480 - auc: 0.6735 - precision: 0.6206 - recall: 0.6307\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6480 - binary_accuracy: 0.6271 - binary_crossentropy: 0.6480 - auc: 0.6737 - precision: 0.6217 - recall: 0.6305\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6483 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6483 - auc: 0.6732 - precision: 0.6220 - recall: 0.6308\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6489 - binary_accuracy: 0.6264 - binary_crossentropy: 0.6489 - auc: 0.6725 - precision: 0.6212 - recall: 0.6302\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6489 - binary_accuracy: 0.6263 - binary_crossentropy: 0.6489 - auc: 0.6724 - precision: 0.6212 - recall: 0.6301\n",
      "11/50 [=====>........................] - ETA: 7s - loss: 0.6488 - binary_accuracy: 0.6266 - binary_crossentropy: 0.6488 - auc: 0.6725 - precision: 0.6215 - recall: 0.6300\n",
      "12/50 [======>.......................] - ETA: 7s - loss: 0.6487 - binary_accuracy: 0.6265 - binary_crossentropy: 0.6487 - auc: 0.6726 - precision: 0.6216 - recall: 0.6300\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6486 - binary_accuracy: 0.6267 - binary_crossentropy: 0.6486 - auc: 0.6728 - precision: 0.6220 - recall: 0.6299\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6487 - binary_accuracy: 0.6266 - binary_crossentropy: 0.6487 - auc: 0.6726 - precision: 0.6218 - recall: 0.6298\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6487 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6487 - auc: 0.6727 - precision: 0.6220 - recall: 0.6302\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6488 - binary_accuracy: 0.6266 - binary_crossentropy: 0.6488 - auc: 0.6725 - precision: 0.6217 - recall: 0.6300\n",
      "17/50 [=========>....................] - ETA: 6s - loss: 0.6489 - binary_accuracy: 0.6265 - binary_crossentropy: 0.6489 - auc: 0.6724 - precision: 0.6216 - recall: 0.6299\n",
      "18/50 [=========>....................] - ETA: 6s - loss: 0.6490 - binary_accuracy: 0.6268 - binary_crossentropy: 0.6490 - auc: 0.6725 - precision: 0.6219 - recall: 0.6300\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6489 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6489 - auc: 0.6726 - precision: 0.6222 - recall: 0.6304\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6489 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6223 - recall: 0.6305\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6488 - binary_accuracy: 0.6271 - binary_crossentropy: 0.6488 - auc: 0.6728 - precision: 0.6221 - recall: 0.6308\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6488 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6488 - auc: 0.6728 - precision: 0.6219 - recall: 0.6308\n",
      "23/50 [============>.................] - ETA: 5s - loss: 0.6488 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6488 - auc: 0.6728 - precision: 0.6220 - recall: 0.6311\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6489 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6221 - recall: 0.6311\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6489 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6222 - recall: 0.6313\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6488 - binary_accuracy: 0.6271 - binary_crossentropy: 0.6488 - auc: 0.6728 - precision: 0.6222 - recall: 0.6315\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6489 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6221 - recall: 0.6317\n",
      "28/50 [===============>..............] - ETA: 4s - loss: 0.6488 - binary_accuracy: 0.6271 - binary_crossentropy: 0.6488 - auc: 0.6728 - precision: 0.6223 - recall: 0.6319\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6488 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6488 - auc: 0.6727 - precision: 0.6223 - recall: 0.6318\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6489 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6223 - recall: 0.6318\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6489 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6222 - recall: 0.6321\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6489 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6489 - auc: 0.6726 - precision: 0.6222 - recall: 0.6322\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6488 - binary_accuracy: 0.6271 - binary_crossentropy: 0.6488 - auc: 0.6728 - precision: 0.6224 - recall: 0.6325\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6489 - binary_accuracy: 0.6271 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6222 - recall: 0.6325\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6489 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6223 - recall: 0.6324\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6490 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6490 - auc: 0.6725 - precision: 0.6222 - recall: 0.6325\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6491 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6491 - auc: 0.6724 - precision: 0.6224 - recall: 0.6326\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6490 - binary_accuracy: 0.6272 - binary_crossentropy: 0.6490 - auc: 0.6726 - precision: 0.6227 - recall: 0.6327\n",
      "39/50 [======================>.......] - ETA: 2s - loss: 0.6488 - binary_accuracy: 0.6274 - binary_crossentropy: 0.6488 - auc: 0.6728 - precision: 0.6229 - recall: 0.6329\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6489 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6229 - recall: 0.6328\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6489 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6229 - recall: 0.6328\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6489 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6229 - recall: 0.6328\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6489 - binary_accuracy: 0.6272 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6229 - recall: 0.6327\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6488 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6488 - auc: 0.6727 - precision: 0.6228 - recall: 0.6329\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6489 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6489 - auc: 0.6727 - precision: 0.6229 - recall: 0.6330\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6488 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6488 - auc: 0.6728 - precision: 0.6231 - recall: 0.6332\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6488 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6488 - auc: 0.6727 - precision: 0.6230 - recall: 0.6332\n",
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6487 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6487 - auc: 0.6728 - precision: 0.6230 - recall: 0.6334\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6487 - binary_accuracy: 0.6274 - binary_crossentropy: 0.6487 - auc: 0.6729 - precision: 0.6230 - recall: 0.6337\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6487 - binary_accuracy: 0.6275 - binary_crossentropy: 0.6487 - auc: 0.6730 - precision: 0.6231 - recall: 0.6339\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 796ms/step - loss: 0.6487 - binary_accuracy: 0.6275 - binary_crossentropy: 0.6487 - auc: 0.6730 - precision: 0.6231 - recall: 0.6339 - val_loss: 0.6493 - val_binary_accuracy: 0.6374 - val_binary_crossentropy: 0.6493 - val_auc: 0.6870 - val_precision: 0.6579 - val_recall: 0.5914\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 8/585\n",
      " 1/50 [..............................] - ETA: 10s - loss: 0.6498 - binary_accuracy: 0.6251 - binary_crossentropy: 0.6498 - auc: 0.6724 - precision: 0.6171 - recall: 0.6364\n",
      " 2/50 [>.............................] - ETA: 8s - loss: 0.6497 - binary_accuracy: 0.6271 - binary_crossentropy: 0.6497 - auc: 0.6725 - precision: 0.6205 - recall: 0.6381 \n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6493 - binary_accuracy: 0.6259 - binary_crossentropy: 0.6493 - auc: 0.6726 - precision: 0.6201 - recall: 0.6349\n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.6487 - binary_accuracy: 0.6263 - binary_crossentropy: 0.6487 - auc: 0.6732 - precision: 0.6207 - recall: 0.6336\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6485 - binary_accuracy: 0.6266 - binary_crossentropy: 0.6485 - auc: 0.6734 - precision: 0.6212 - recall: 0.6319\n",
      " 6/50 [==>...........................] - ETA: 8s - loss: 0.6483 - binary_accuracy: 0.6265 - binary_crossentropy: 0.6483 - auc: 0.6736 - precision: 0.6216 - recall: 0.6302\n",
      " 7/50 [===>..........................] - ETA: 8s - loss: 0.6481 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6481 - auc: 0.6738 - precision: 0.6222 - recall: 0.6298\n",
      " 8/50 [===>..........................] - ETA: 8s - loss: 0.6479 - binary_accuracy: 0.6275 - binary_crossentropy: 0.6479 - auc: 0.6742 - precision: 0.6236 - recall: 0.6300\n",
      " 9/50 [====>.........................] - ETA: 8s - loss: 0.6480 - binary_accuracy: 0.6276 - binary_crossentropy: 0.6480 - auc: 0.6741 - precision: 0.6235 - recall: 0.6299\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6482 - binary_accuracy: 0.6275 - binary_crossentropy: 0.6482 - auc: 0.6738 - precision: 0.6239 - recall: 0.6298\n",
      "11/50 [=====>........................] - ETA: 7s - loss: 0.6483 - binary_accuracy: 0.6274 - binary_crossentropy: 0.6483 - auc: 0.6738 - precision: 0.6237 - recall: 0.6300\n",
      "12/50 [======>.......................] - ETA: 7s - loss: 0.6482 - binary_accuracy: 0.6275 - binary_crossentropy: 0.6482 - auc: 0.6739 - precision: 0.6239 - recall: 0.6303\n",
      "13/50 [======>.......................] - ETA: 7s - loss: 0.6481 - binary_accuracy: 0.6276 - binary_crossentropy: 0.6481 - auc: 0.6739 - precision: 0.6238 - recall: 0.6304\n",
      "14/50 [=======>......................] - ETA: 7s - loss: 0.6485 - binary_accuracy: 0.6271 - binary_crossentropy: 0.6485 - auc: 0.6733 - precision: 0.6234 - recall: 0.6304\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6486 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6486 - auc: 0.6730 - precision: 0.6232 - recall: 0.6303\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6490 - binary_accuracy: 0.6262 - binary_crossentropy: 0.6490 - auc: 0.6722 - precision: 0.6225 - recall: 0.6298\n",
      "17/50 [=========>....................] - ETA: 6s - loss: 0.6492 - binary_accuracy: 0.6258 - binary_crossentropy: 0.6492 - auc: 0.6720 - precision: 0.6219 - recall: 0.6299\n",
      "18/50 [=========>....................] - ETA: 6s - loss: 0.6489 - binary_accuracy: 0.6261 - binary_crossentropy: 0.6489 - auc: 0.6724 - precision: 0.6219 - recall: 0.6309\n",
      "19/50 [==========>...................] - ETA: 6s - loss: 0.6488 - binary_accuracy: 0.6262 - binary_crossentropy: 0.6488 - auc: 0.6726 - precision: 0.6217 - recall: 0.6315\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6488 - binary_accuracy: 0.6263 - binary_crossentropy: 0.6488 - auc: 0.6726 - precision: 0.6216 - recall: 0.6322\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6488 - binary_accuracy: 0.6263 - binary_crossentropy: 0.6488 - auc: 0.6726 - precision: 0.6214 - recall: 0.6328\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6488 - binary_accuracy: 0.6265 - binary_crossentropy: 0.6488 - auc: 0.6726 - precision: 0.6212 - recall: 0.6335\n",
      "23/50 [============>.................] - ETA: 5s - loss: 0.6487 - binary_accuracy: 0.6268 - binary_crossentropy: 0.6487 - auc: 0.6728 - precision: 0.6216 - recall: 0.6339\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6485 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6485 - auc: 0.6731 - precision: 0.6219 - recall: 0.6340\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6485 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6485 - auc: 0.6730 - precision: 0.6221 - recall: 0.6338\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6486 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6486 - auc: 0.6729 - precision: 0.6221 - recall: 0.6336\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6484 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6484 - auc: 0.6732 - precision: 0.6224 - recall: 0.6335\n",
      "28/50 [===============>..............] - ETA: 4s - loss: 0.6485 - binary_accuracy: 0.6268 - binary_crossentropy: 0.6485 - auc: 0.6730 - precision: 0.6223 - recall: 0.6333\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6485 - binary_accuracy: 0.6268 - binary_crossentropy: 0.6485 - auc: 0.6730 - precision: 0.6223 - recall: 0.6333\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6485 - binary_accuracy: 0.6268 - binary_crossentropy: 0.6485 - auc: 0.6730 - precision: 0.6224 - recall: 0.6333\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6484 - binary_accuracy: 0.6270 - binary_crossentropy: 0.6484 - auc: 0.6732 - precision: 0.6225 - recall: 0.6336\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6485 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6485 - auc: 0.6730 - precision: 0.6226 - recall: 0.6336\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6485 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6485 - auc: 0.6731 - precision: 0.6226 - recall: 0.6338\n",
      "34/50 [===================>..........] - ETA: 3s - loss: 0.6484 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6484 - auc: 0.6732 - precision: 0.6225 - recall: 0.6341\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6484 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6484 - auc: 0.6732 - precision: 0.6225 - recall: 0.6344\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6485 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6485 - auc: 0.6731 - precision: 0.6225 - recall: 0.6344\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6484 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6484 - auc: 0.6732 - precision: 0.6225 - recall: 0.6345\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6485 - binary_accuracy: 0.6267 - binary_crossentropy: 0.6485 - auc: 0.6731 - precision: 0.6224 - recall: 0.6344\n",
      "39/50 [======================>.......] - ETA: 2s - loss: 0.6485 - binary_accuracy: 0.6268 - binary_crossentropy: 0.6485 - auc: 0.6731 - precision: 0.6224 - recall: 0.6345\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6485 - binary_accuracy: 0.6267 - binary_crossentropy: 0.6485 - auc: 0.6731 - precision: 0.6224 - recall: 0.6345\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6485 - binary_accuracy: 0.6268 - binary_crossentropy: 0.6485 - auc: 0.6731 - precision: 0.6224 - recall: 0.6345\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6485 - binary_accuracy: 0.6266 - binary_crossentropy: 0.6485 - auc: 0.6730 - precision: 0.6222 - recall: 0.6345\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6485 - binary_accuracy: 0.6266 - binary_crossentropy: 0.6485 - auc: 0.6730 - precision: 0.6220 - recall: 0.6345\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6486 - binary_accuracy: 0.6266 - binary_crossentropy: 0.6486 - auc: 0.6729 - precision: 0.6218 - recall: 0.6346\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6486 - binary_accuracy: 0.6266 - binary_crossentropy: 0.6486 - auc: 0.6730 - precision: 0.6220 - recall: 0.6347\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6486 - binary_accuracy: 0.6267 - binary_crossentropy: 0.6486 - auc: 0.6730 - precision: 0.6220 - recall: 0.6348\n",
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6485 - binary_accuracy: 0.6267 - binary_crossentropy: 0.6485 - auc: 0.6730 - precision: 0.6221 - recall: 0.6347\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6485 - binary_accuracy: 0.6266 - binary_crossentropy: 0.6485 - auc: 0.6730 - precision: 0.6221 - recall: 0.6346\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6485 - binary_accuracy: 0.6266 - binary_crossentropy: 0.6485 - auc: 0.6730 - precision: 0.6221 - recall: 0.6345\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6485 - binary_accuracy: 0.6267 - binary_crossentropy: 0.6485 - auc: 0.6731 - precision: 0.6222 - recall: 0.6345\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 791ms/step - loss: 0.6485 - binary_accuracy: 0.6267 - binary_crossentropy: 0.6485 - auc: 0.6731 - precision: 0.6222 - recall: 0.6345 - val_loss: 0.6431 - val_binary_accuracy: 0.6450 - val_binary_crossentropy: 0.6431 - val_auc: 0.6933 - val_precision: 0.6607 - val_recall: 0.6144\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 9/585\n",
      " 1/50 [..............................] - ETA: 12s - loss: 0.6463 - binary_accuracy: 0.6300 - binary_crossentropy: 0.6463 - auc: 0.6757 - precision: 0.6237 - recall: 0.6336\n",
      " 2/50 [>.............................] - ETA: 9s - loss: 0.6453 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6453 - auc: 0.6776 - precision: 0.6251 - recall: 0.6350 \n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6468 - binary_accuracy: 0.6290 - binary_crossentropy: 0.6468 - auc: 0.6756 - precision: 0.6245 - recall: 0.6313\n",
      " 4/50 [=>............................] - ETA: 7s - loss: 0.6472 - binary_accuracy: 0.6289 - binary_crossentropy: 0.6472 - auc: 0.6752 - precision: 0.6249 - recall: 0.6320\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6485 - binary_accuracy: 0.6275 - binary_crossentropy: 0.6485 - auc: 0.6733 - precision: 0.6237 - recall: 0.6312\n",
      " 6/50 [==>...........................] - ETA: 7s - loss: 0.6484 - binary_accuracy: 0.6275 - binary_crossentropy: 0.6484 - auc: 0.6732 - precision: 0.6236 - recall: 0.6317\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6483 - binary_accuracy: 0.6275 - binary_crossentropy: 0.6483 - auc: 0.6732 - precision: 0.6239 - recall: 0.6321\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6486 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6486 - auc: 0.6727 - precision: 0.6232 - recall: 0.6317\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6480 - binary_accuracy: 0.6271 - binary_crossentropy: 0.6480 - auc: 0.6733 - precision: 0.6233 - recall: 0.6322\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6478 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6478 - auc: 0.6736 - precision: 0.6234 - recall: 0.6327\n",
      "11/50 [=====>........................] - ETA: 6s - loss: 0.6480 - binary_accuracy: 0.6274 - binary_crossentropy: 0.6480 - auc: 0.6734 - precision: 0.6232 - recall: 0.6333\n",
      "12/50 [======>.......................] - ETA: 6s - loss: 0.6480 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6480 - auc: 0.6735 - precision: 0.6226 - recall: 0.6336\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6477 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6477 - auc: 0.6740 - precision: 0.6230 - recall: 0.6346\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6476 - binary_accuracy: 0.6283 - binary_crossentropy: 0.6476 - auc: 0.6742 - precision: 0.6239 - recall: 0.6351\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6478 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6478 - auc: 0.6740 - precision: 0.6236 - recall: 0.6352\n",
      "16/50 [========>.....................] - ETA: 5s - loss: 0.6478 - binary_accuracy: 0.6283 - binary_crossentropy: 0.6478 - auc: 0.6741 - precision: 0.6236 - recall: 0.6354\n",
      "17/50 [=========>....................] - ETA: 5s - loss: 0.6477 - binary_accuracy: 0.6283 - binary_crossentropy: 0.6477 - auc: 0.6741 - precision: 0.6234 - recall: 0.6356\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6475 - binary_accuracy: 0.6284 - binary_crossentropy: 0.6475 - auc: 0.6744 - precision: 0.6239 - recall: 0.6356\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6476 - binary_accuracy: 0.6284 - binary_crossentropy: 0.6476 - auc: 0.6742 - precision: 0.6238 - recall: 0.6357\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6476 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6476 - auc: 0.6742 - precision: 0.6234 - recall: 0.6356\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6477 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6477 - auc: 0.6741 - precision: 0.6230 - recall: 0.6355\n",
      "22/50 [============>.................] - ETA: 4s - loss: 0.6476 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6476 - auc: 0.6741 - precision: 0.6232 - recall: 0.6356\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6476 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6476 - auc: 0.6741 - precision: 0.6231 - recall: 0.6354\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6477 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6477 - auc: 0.6740 - precision: 0.6231 - recall: 0.6353\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6477 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6477 - auc: 0.6740 - precision: 0.6232 - recall: 0.6353\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6477 - binary_accuracy: 0.6280 - binary_crossentropy: 0.6477 - auc: 0.6740 - precision: 0.6229 - recall: 0.6352\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6477 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6477 - auc: 0.6740 - precision: 0.6227 - recall: 0.6349\n",
      "28/50 [===============>..............] - ETA: 3s - loss: 0.6478 - binary_accuracy: 0.6278 - binary_crossentropy: 0.6478 - auc: 0.6739 - precision: 0.6227 - recall: 0.6346\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6478 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6478 - auc: 0.6739 - precision: 0.6229 - recall: 0.6345\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6478 - binary_accuracy: 0.6278 - binary_crossentropy: 0.6478 - auc: 0.6738 - precision: 0.6228 - recall: 0.6344\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6479 - binary_accuracy: 0.6277 - binary_crossentropy: 0.6479 - auc: 0.6737 - precision: 0.6229 - recall: 0.6343\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6478 - binary_accuracy: 0.6278 - binary_crossentropy: 0.6478 - auc: 0.6738 - precision: 0.6229 - recall: 0.6345\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6478 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6478 - auc: 0.6739 - precision: 0.6230 - recall: 0.6346\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6478 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6478 - auc: 0.6738 - precision: 0.6230 - recall: 0.6345\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6478 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6478 - auc: 0.6738 - precision: 0.6231 - recall: 0.6345\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6478 - binary_accuracy: 0.6278 - binary_crossentropy: 0.6478 - auc: 0.6738 - precision: 0.6231 - recall: 0.6345\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6479 - binary_accuracy: 0.6278 - binary_crossentropy: 0.6479 - auc: 0.6737 - precision: 0.6230 - recall: 0.6347\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6478 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6478 - auc: 0.6738 - precision: 0.6231 - recall: 0.6349\n",
      "39/50 [======================>.......] - ETA: 1s - loss: 0.6477 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6477 - auc: 0.6739 - precision: 0.6232 - recall: 0.6350\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6476 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6476 - auc: 0.6741 - precision: 0.6234 - recall: 0.6355\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6475 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6475 - auc: 0.6742 - precision: 0.6234 - recall: 0.6357\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6475 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6475 - auc: 0.6743 - precision: 0.6232 - recall: 0.6360\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6475 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6475 - auc: 0.6743 - precision: 0.6232 - recall: 0.6361\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6474 - binary_accuracy: 0.6283 - binary_crossentropy: 0.6474 - auc: 0.6744 - precision: 0.6234 - recall: 0.6363\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6475 - binary_accuracy: 0.6284 - binary_crossentropy: 0.6475 - auc: 0.6744 - precision: 0.6234 - recall: 0.6363\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6474 - binary_accuracy: 0.6285 - binary_crossentropy: 0.6474 - auc: 0.6745 - precision: 0.6235 - recall: 0.6364\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6474 - binary_accuracy: 0.6285 - binary_crossentropy: 0.6474 - auc: 0.6746 - precision: 0.6236 - recall: 0.6364\n",
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6475 - binary_accuracy: 0.6284 - binary_crossentropy: 0.6475 - auc: 0.6744 - precision: 0.6235 - recall: 0.6363\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6475 - binary_accuracy: 0.6284 - binary_crossentropy: 0.6475 - auc: 0.6743 - precision: 0.6235 - recall: 0.6362\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6476 - binary_accuracy: 0.6284 - binary_crossentropy: 0.6476 - auc: 0.6743 - precision: 0.6234 - recall: 0.6361\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 797ms/step - loss: 0.6476 - binary_accuracy: 0.6284 - binary_crossentropy: 0.6476 - auc: 0.6743 - precision: 0.6234 - recall: 0.6361 - val_loss: 0.6390 - val_binary_accuracy: 0.6486 - val_binary_crossentropy: 0.6390 - val_auc: 0.6952 - val_precision: 0.6586 - val_recall: 0.6355\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 10/585\n",
      " 1/50 [..............................] - ETA: 11s - loss: 0.6495 - binary_accuracy: 0.6244 - binary_crossentropy: 0.6495 - auc: 0.6705 - precision: 0.6233 - recall: 0.6261\n",
      " 2/50 [>.............................] - ETA: 8s - loss: 0.6471 - binary_accuracy: 0.6253 - binary_crossentropy: 0.6471 - auc: 0.6734 - precision: 0.6239 - recall: 0.6267 \n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6478 - binary_accuracy: 0.6247 - binary_crossentropy: 0.6478 - auc: 0.6728 - precision: 0.6233 - recall: 0.6275\n",
      " 4/50 [=>............................] - ETA: 7s - loss: 0.6488 - binary_accuracy: 0.6243 - binary_crossentropy: 0.6488 - auc: 0.6718 - precision: 0.6229 - recall: 0.6279\n",
      " 5/50 [==>...........................] - ETA: 7s - loss: 0.6490 - binary_accuracy: 0.6242 - binary_crossentropy: 0.6490 - auc: 0.6717 - precision: 0.6229 - recall: 0.6287\n",
      " 6/50 [==>...........................] - ETA: 7s - loss: 0.6491 - binary_accuracy: 0.6244 - binary_crossentropy: 0.6491 - auc: 0.6716 - precision: 0.6223 - recall: 0.6294\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6483 - binary_accuracy: 0.6257 - binary_crossentropy: 0.6483 - auc: 0.6730 - precision: 0.6234 - recall: 0.6314\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6484 - binary_accuracy: 0.6260 - binary_crossentropy: 0.6484 - auc: 0.6729 - precision: 0.6234 - recall: 0.6321\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6488 - binary_accuracy: 0.6261 - binary_crossentropy: 0.6488 - auc: 0.6724 - precision: 0.6237 - recall: 0.6325\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6486 - binary_accuracy: 0.6266 - binary_crossentropy: 0.6486 - auc: 0.6727 - precision: 0.6240 - recall: 0.6335\n",
      "11/50 [=====>........................] - ETA: 7s - loss: 0.6483 - binary_accuracy: 0.6269 - binary_crossentropy: 0.6483 - auc: 0.6730 - precision: 0.6237 - recall: 0.6341\n",
      "12/50 [======>.......................] - ETA: 6s - loss: 0.6479 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6479 - auc: 0.6734 - precision: 0.6236 - recall: 0.6350\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6479 - binary_accuracy: 0.6272 - binary_crossentropy: 0.6479 - auc: 0.6733 - precision: 0.6234 - recall: 0.6354\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6477 - binary_accuracy: 0.6278 - binary_crossentropy: 0.6477 - auc: 0.6737 - precision: 0.6237 - recall: 0.6363\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6474 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6474 - auc: 0.6740 - precision: 0.6239 - recall: 0.6364\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6478 - binary_accuracy: 0.6275 - binary_crossentropy: 0.6478 - auc: 0.6736 - precision: 0.6235 - recall: 0.6361\n",
      "17/50 [=========>....................] - ETA: 6s - loss: 0.6479 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6479 - auc: 0.6735 - precision: 0.6235 - recall: 0.6359\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6480 - binary_accuracy: 0.6272 - binary_crossentropy: 0.6480 - auc: 0.6733 - precision: 0.6230 - recall: 0.6359\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6479 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6479 - auc: 0.6735 - precision: 0.6230 - recall: 0.6360\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6478 - binary_accuracy: 0.6274 - binary_crossentropy: 0.6478 - auc: 0.6736 - precision: 0.6232 - recall: 0.6362\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6479 - binary_accuracy: 0.6273 - binary_crossentropy: 0.6479 - auc: 0.6735 - precision: 0.6232 - recall: 0.6362\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6477 - binary_accuracy: 0.6276 - binary_crossentropy: 0.6477 - auc: 0.6737 - precision: 0.6237 - recall: 0.6364\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6478 - binary_accuracy: 0.6274 - binary_crossentropy: 0.6478 - auc: 0.6735 - precision: 0.6231 - recall: 0.6365\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6478 - binary_accuracy: 0.6275 - binary_crossentropy: 0.6478 - auc: 0.6736 - precision: 0.6233 - recall: 0.6366\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6476 - binary_accuracy: 0.6276 - binary_crossentropy: 0.6476 - auc: 0.6738 - precision: 0.6233 - recall: 0.6363\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6476 - binary_accuracy: 0.6276 - binary_crossentropy: 0.6476 - auc: 0.6738 - precision: 0.6234 - recall: 0.6363\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6475 - binary_accuracy: 0.6278 - binary_crossentropy: 0.6475 - auc: 0.6740 - precision: 0.6234 - recall: 0.6362\n",
      "28/50 [===============>..............] - ETA: 3s - loss: 0.6475 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6475 - auc: 0.6741 - precision: 0.6235 - recall: 0.6362\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6474 - binary_accuracy: 0.6280 - binary_crossentropy: 0.6474 - auc: 0.6742 - precision: 0.6235 - recall: 0.6362\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6474 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6474 - auc: 0.6742 - precision: 0.6236 - recall: 0.6360\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6474 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6474 - auc: 0.6742 - precision: 0.6235 - recall: 0.6359\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6474 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6474 - auc: 0.6742 - precision: 0.6237 - recall: 0.6359\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6474 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6474 - auc: 0.6743 - precision: 0.6238 - recall: 0.6356\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6474 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6474 - auc: 0.6743 - precision: 0.6239 - recall: 0.6357\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6473 - binary_accuracy: 0.6283 - binary_crossentropy: 0.6473 - auc: 0.6745 - precision: 0.6240 - recall: 0.6358\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6474 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6474 - auc: 0.6743 - precision: 0.6239 - recall: 0.6357\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6473 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6473 - auc: 0.6744 - precision: 0.6239 - recall: 0.6357\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6473 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6473 - auc: 0.6744 - precision: 0.6238 - recall: 0.6358\n",
      "39/50 [======================>.......] - ETA: 1s - loss: 0.6474 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6474 - auc: 0.6743 - precision: 0.6236 - recall: 0.6356\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6474 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6474 - auc: 0.6743 - precision: 0.6235 - recall: 0.6355\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6474 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6474 - auc: 0.6744 - precision: 0.6238 - recall: 0.6354\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6474 - binary_accuracy: 0.6280 - binary_crossentropy: 0.6474 - auc: 0.6743 - precision: 0.6237 - recall: 0.6353\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6475 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6475 - auc: 0.6742 - precision: 0.6236 - recall: 0.6353\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6474 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6474 - auc: 0.6742 - precision: 0.6236 - recall: 0.6355\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6473 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6473 - auc: 0.6744 - precision: 0.6238 - recall: 0.6358\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6472 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6472 - auc: 0.6745 - precision: 0.6238 - recall: 0.6362\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6471 - binary_accuracy: 0.6283 - binary_crossentropy: 0.6471 - auc: 0.6747 - precision: 0.6240 - recall: 0.6364\n",
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6472 - binary_accuracy: 0.6284 - binary_crossentropy: 0.6472 - auc: 0.6747 - precision: 0.6239 - recall: 0.6367\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6471 - binary_accuracy: 0.6285 - binary_crossentropy: 0.6471 - auc: 0.6747 - precision: 0.6239 - recall: 0.6370\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6472 - binary_accuracy: 0.6285 - binary_crossentropy: 0.6472 - auc: 0.6747 - precision: 0.6240 - recall: 0.6371\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 41s 834ms/step - loss: 0.6472 - binary_accuracy: 0.6285 - binary_crossentropy: 0.6472 - auc: 0.6747 - precision: 0.6240 - recall: 0.6371 - val_loss: 0.6350 - val_binary_accuracy: 0.6503 - val_binary_crossentropy: 0.6350 - val_auc: 0.6979 - val_precision: 0.6633 - val_recall: 0.6283\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 11/585\n",
      " 1/50 [..............................] - ETA: 10s - loss: 0.6472 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6472 - auc: 0.6751 - precision: 0.6209 - recall: 0.6417\n",
      " 2/50 [>.............................] - ETA: 8s - loss: 0.6467 - binary_accuracy: 0.6291 - binary_crossentropy: 0.6467 - auc: 0.6758 - precision: 0.6234 - recall: 0.6420 \n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6466 - binary_accuracy: 0.6293 - binary_crossentropy: 0.6466 - auc: 0.6759 - precision: 0.6235 - recall: 0.6428\n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.6473 - binary_accuracy: 0.6294 - binary_crossentropy: 0.6473 - auc: 0.6750 - precision: 0.6256 - recall: 0.6416\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6476 - binary_accuracy: 0.6286 - binary_crossentropy: 0.6476 - auc: 0.6744 - precision: 0.6245 - recall: 0.6404\n",
      " 6/50 [==>...........................] - ETA: 7s - loss: 0.6479 - binary_accuracy: 0.6283 - binary_crossentropy: 0.6479 - auc: 0.6737 - precision: 0.6240 - recall: 0.6396\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6485 - binary_accuracy: 0.6272 - binary_crossentropy: 0.6485 - auc: 0.6726 - precision: 0.6224 - recall: 0.6385\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6480 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6480 - auc: 0.6733 - precision: 0.6235 - recall: 0.6388\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6479 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6479 - auc: 0.6734 - precision: 0.6236 - recall: 0.6385\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6481 - binary_accuracy: 0.6275 - binary_crossentropy: 0.6481 - auc: 0.6730 - precision: 0.6229 - recall: 0.6384\n",
      "11/50 [=====>........................] - ETA: 6s - loss: 0.6477 - binary_accuracy: 0.6280 - binary_crossentropy: 0.6477 - auc: 0.6737 - precision: 0.6229 - recall: 0.6391\n",
      "12/50 [======>.......................] - ETA: 6s - loss: 0.6476 - binary_accuracy: 0.6280 - binary_crossentropy: 0.6476 - auc: 0.6739 - precision: 0.6229 - recall: 0.6392\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6477 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6477 - auc: 0.6737 - precision: 0.6228 - recall: 0.6393\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6478 - binary_accuracy: 0.6279 - binary_crossentropy: 0.6478 - auc: 0.6736 - precision: 0.6228 - recall: 0.6387\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6476 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6476 - auc: 0.6738 - precision: 0.6231 - recall: 0.6387\n",
      "16/50 [========>.....................] - ETA: 5s - loss: 0.6476 - binary_accuracy: 0.6283 - binary_crossentropy: 0.6476 - auc: 0.6738 - precision: 0.6234 - recall: 0.6379\n",
      "17/50 [=========>....................] - ETA: 5s - loss: 0.6476 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6476 - auc: 0.6737 - precision: 0.6234 - recall: 0.6370\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6476 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6476 - auc: 0.6737 - precision: 0.6236 - recall: 0.6367\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6477 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6477 - auc: 0.6736 - precision: 0.6235 - recall: 0.6362\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6478 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6478 - auc: 0.6734 - precision: 0.6234 - recall: 0.6359\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6477 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6477 - auc: 0.6735 - precision: 0.6235 - recall: 0.6357\n",
      "22/50 [============>.................] - ETA: 4s - loss: 0.6476 - binary_accuracy: 0.6283 - binary_crossentropy: 0.6476 - auc: 0.6737 - precision: 0.6238 - recall: 0.6359\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6475 - binary_accuracy: 0.6284 - binary_crossentropy: 0.6475 - auc: 0.6738 - precision: 0.6238 - recall: 0.6359\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6476 - binary_accuracy: 0.6281 - binary_crossentropy: 0.6476 - auc: 0.6737 - precision: 0.6237 - recall: 0.6356\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6476 - binary_accuracy: 0.6282 - binary_crossentropy: 0.6476 - auc: 0.6738 - precision: 0.6238 - recall: 0.6357\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6474 - binary_accuracy: 0.6284 - binary_crossentropy: 0.6474 - auc: 0.6740 - precision: 0.6239 - recall: 0.6360\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6473 - binary_accuracy: 0.6286 - binary_crossentropy: 0.6473 - auc: 0.6742 - precision: 0.6240 - recall: 0.6364\n",
      "28/50 [===============>..............] - ETA: 3s - loss: 0.6471 - binary_accuracy: 0.6288 - binary_crossentropy: 0.6471 - auc: 0.6745 - precision: 0.6240 - recall: 0.6367\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6471 - binary_accuracy: 0.6290 - binary_crossentropy: 0.6471 - auc: 0.6746 - precision: 0.6241 - recall: 0.6371\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6470 - binary_accuracy: 0.6291 - binary_crossentropy: 0.6470 - auc: 0.6747 - precision: 0.6242 - recall: 0.6374\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6469 - binary_accuracy: 0.6291 - binary_crossentropy: 0.6469 - auc: 0.6748 - precision: 0.6241 - recall: 0.6375\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6469 - binary_accuracy: 0.6292 - binary_crossentropy: 0.6469 - auc: 0.6749 - precision: 0.6243 - recall: 0.6377\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6468 - binary_accuracy: 0.6293 - binary_crossentropy: 0.6468 - auc: 0.6750 - precision: 0.6243 - recall: 0.6378\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6468 - binary_accuracy: 0.6294 - binary_crossentropy: 0.6468 - auc: 0.6750 - precision: 0.6244 - recall: 0.6381\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6468 - binary_accuracy: 0.6294 - binary_crossentropy: 0.6468 - auc: 0.6749 - precision: 0.6243 - recall: 0.6381\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6468 - binary_accuracy: 0.6293 - binary_crossentropy: 0.6468 - auc: 0.6749 - precision: 0.6242 - recall: 0.6381\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6468 - binary_accuracy: 0.6293 - binary_crossentropy: 0.6468 - auc: 0.6749 - precision: 0.6243 - recall: 0.6381\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6467 - binary_accuracy: 0.6294 - binary_crossentropy: 0.6467 - auc: 0.6750 - precision: 0.6245 - recall: 0.6381\n",
      "39/50 [======================>.......] - ETA: 1s - loss: 0.6467 - binary_accuracy: 0.6294 - binary_crossentropy: 0.6467 - auc: 0.6751 - precision: 0.6246 - recall: 0.6380\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6466 - binary_accuracy: 0.6295 - binary_crossentropy: 0.6466 - auc: 0.6752 - precision: 0.6246 - recall: 0.6381\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6466 - binary_accuracy: 0.6296 - binary_crossentropy: 0.6466 - auc: 0.6753 - precision: 0.6248 - recall: 0.6383\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6465 - binary_accuracy: 0.6297 - binary_crossentropy: 0.6465 - auc: 0.6755 - precision: 0.6248 - recall: 0.6384\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6465 - binary_accuracy: 0.6296 - binary_crossentropy: 0.6465 - auc: 0.6755 - precision: 0.6248 - recall: 0.6385\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6465 - binary_accuracy: 0.6297 - binary_crossentropy: 0.6465 - auc: 0.6755 - precision: 0.6248 - recall: 0.6387\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6465 - binary_accuracy: 0.6296 - binary_crossentropy: 0.6465 - auc: 0.6754 - precision: 0.6248 - recall: 0.6387\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6465 - binary_accuracy: 0.6295 - binary_crossentropy: 0.6465 - auc: 0.6754 - precision: 0.6247 - recall: 0.6386\n",
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6465 - binary_accuracy: 0.6295 - binary_crossentropy: 0.6465 - auc: 0.6754 - precision: 0.6247 - recall: 0.6386\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6466 - binary_accuracy: 0.6294 - binary_crossentropy: 0.6466 - auc: 0.6753 - precision: 0.6247 - recall: 0.6385\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6465 - binary_accuracy: 0.6294 - binary_crossentropy: 0.6465 - auc: 0.6754 - precision: 0.6246 - recall: 0.6387\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6465 - binary_accuracy: 0.6294 - binary_crossentropy: 0.6465 - auc: 0.6754 - precision: 0.6247 - recall: 0.6387\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 795ms/step - loss: 0.6465 - binary_accuracy: 0.6294 - binary_crossentropy: 0.6465 - auc: 0.6754 - precision: 0.6247 - recall: 0.6387 - val_loss: 0.6313 - val_binary_accuracy: 0.6546 - val_binary_crossentropy: 0.6313 - val_auc: 0.7026 - val_precision: 0.6652 - val_recall: 0.6401\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 12/585\n",
      " 1/50 [..............................] - ETA: 10s - loss: 0.6475 - binary_accuracy: 0.6250 - binary_crossentropy: 0.6475 - auc: 0.6734 - precision: 0.6206 - recall: 0.6374\n",
      " 2/50 [>.............................] - ETA: 7s - loss: 0.6481 - binary_accuracy: 0.6259 - binary_crossentropy: 0.6481 - auc: 0.6731 - precision: 0.6200 - recall: 0.6382 \n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6454 - binary_accuracy: 0.6291 - binary_crossentropy: 0.6454 - auc: 0.6772 - precision: 0.6235 - recall: 0.6396\n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.6462 - binary_accuracy: 0.6289 - binary_crossentropy: 0.6462 - auc: 0.6765 - precision: 0.6237 - recall: 0.6402\n",
      " 5/50 [==>...........................] - ETA: 7s - loss: 0.6466 - binary_accuracy: 0.6289 - binary_crossentropy: 0.6466 - auc: 0.6758 - precision: 0.6242 - recall: 0.6400\n",
      " 6/50 [==>...........................] - ETA: 7s - loss: 0.6463 - binary_accuracy: 0.6297 - binary_crossentropy: 0.6463 - auc: 0.6763 - precision: 0.6249 - recall: 0.6407\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6466 - binary_accuracy: 0.6296 - binary_crossentropy: 0.6466 - auc: 0.6758 - precision: 0.6252 - recall: 0.6403\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6462 - binary_accuracy: 0.6297 - binary_crossentropy: 0.6462 - auc: 0.6762 - precision: 0.6257 - recall: 0.6396\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6461 - binary_accuracy: 0.6296 - binary_crossentropy: 0.6461 - auc: 0.6763 - precision: 0.6260 - recall: 0.6395\n",
      "10/50 [=====>........................] - ETA: 6s - loss: 0.6457 - binary_accuracy: 0.6302 - binary_crossentropy: 0.6457 - auc: 0.6770 - precision: 0.6266 - recall: 0.6403\n",
      "11/50 [=====>........................] - ETA: 6s - loss: 0.6457 - binary_accuracy: 0.6304 - binary_crossentropy: 0.6457 - auc: 0.6770 - precision: 0.6261 - recall: 0.6406\n",
      "12/50 [======>.......................] - ETA: 6s - loss: 0.6457 - binary_accuracy: 0.6303 - binary_crossentropy: 0.6457 - auc: 0.6769 - precision: 0.6252 - recall: 0.6405\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6456 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6456 - auc: 0.6772 - precision: 0.6254 - recall: 0.6407\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6458 - binary_accuracy: 0.6302 - binary_crossentropy: 0.6458 - auc: 0.6769 - precision: 0.6253 - recall: 0.6400\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6459 - binary_accuracy: 0.6299 - binary_crossentropy: 0.6459 - auc: 0.6766 - precision: 0.6249 - recall: 0.6392\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6457 - binary_accuracy: 0.6300 - binary_crossentropy: 0.6457 - auc: 0.6768 - precision: 0.6250 - recall: 0.6389\n",
      "17/50 [=========>....................] - ETA: 5s - loss: 0.6457 - binary_accuracy: 0.6299 - binary_crossentropy: 0.6457 - auc: 0.6768 - precision: 0.6256 - recall: 0.6380\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6456 - binary_accuracy: 0.6301 - binary_crossentropy: 0.6456 - auc: 0.6770 - precision: 0.6260 - recall: 0.6378\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6456 - binary_accuracy: 0.6301 - binary_crossentropy: 0.6456 - auc: 0.6770 - precision: 0.6258 - recall: 0.6378\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6456 - binary_accuracy: 0.6302 - binary_crossentropy: 0.6456 - auc: 0.6770 - precision: 0.6259 - recall: 0.6379\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6455 - binary_accuracy: 0.6303 - binary_crossentropy: 0.6455 - auc: 0.6771 - precision: 0.6261 - recall: 0.6380\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6456 - binary_accuracy: 0.6304 - binary_crossentropy: 0.6456 - auc: 0.6770 - precision: 0.6260 - recall: 0.6383\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6456 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6456 - auc: 0.6770 - precision: 0.6260 - recall: 0.6384\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6456 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6456 - auc: 0.6770 - precision: 0.6258 - recall: 0.6386\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6456 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6456 - auc: 0.6769 - precision: 0.6260 - recall: 0.6389\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6456 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6456 - auc: 0.6769 - precision: 0.6260 - recall: 0.6388\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6455 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6455 - auc: 0.6770 - precision: 0.6260 - recall: 0.6390\n",
      "28/50 [===============>..............] - ETA: 3s - loss: 0.6456 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6456 - auc: 0.6770 - precision: 0.6260 - recall: 0.6391\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6457 - binary_accuracy: 0.6304 - binary_crossentropy: 0.6457 - auc: 0.6768 - precision: 0.6257 - recall: 0.6391\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6457 - binary_accuracy: 0.6303 - binary_crossentropy: 0.6457 - auc: 0.6767 - precision: 0.6256 - recall: 0.6391\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6457 - binary_accuracy: 0.6303 - binary_crossentropy: 0.6457 - auc: 0.6767 - precision: 0.6257 - recall: 0.6392\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6458 - binary_accuracy: 0.6302 - binary_crossentropy: 0.6458 - auc: 0.6766 - precision: 0.6253 - recall: 0.6392\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6458 - binary_accuracy: 0.6303 - binary_crossentropy: 0.6458 - auc: 0.6767 - precision: 0.6253 - recall: 0.6393\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6457 - binary_accuracy: 0.6304 - binary_crossentropy: 0.6457 - auc: 0.6767 - precision: 0.6253 - recall: 0.6394\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6456 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6456 - auc: 0.6769 - precision: 0.6254 - recall: 0.6396\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6457 - binary_accuracy: 0.6304 - binary_crossentropy: 0.6457 - auc: 0.6768 - precision: 0.6254 - recall: 0.6394\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6456 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6456 - auc: 0.6770 - precision: 0.6255 - recall: 0.6394\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6456 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6456 - auc: 0.6769 - precision: 0.6257 - recall: 0.6391\n",
      "39/50 [======================>.......] - ETA: 1s - loss: 0.6456 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6456 - auc: 0.6769 - precision: 0.6257 - recall: 0.6389\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6455 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6455 - auc: 0.6770 - precision: 0.6257 - recall: 0.6386\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6454 - binary_accuracy: 0.6307 - binary_crossentropy: 0.6454 - auc: 0.6771 - precision: 0.6259 - recall: 0.6385\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6454 - binary_accuracy: 0.6308 - binary_crossentropy: 0.6454 - auc: 0.6772 - precision: 0.6261 - recall: 0.6387\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6454 - binary_accuracy: 0.6307 - binary_crossentropy: 0.6454 - auc: 0.6772 - precision: 0.6261 - recall: 0.6387\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6453 - binary_accuracy: 0.6309 - binary_crossentropy: 0.6453 - auc: 0.6774 - precision: 0.6262 - recall: 0.6389\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6453 - binary_accuracy: 0.6309 - binary_crossentropy: 0.6453 - auc: 0.6773 - precision: 0.6261 - recall: 0.6389\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6454 - binary_accuracy: 0.6308 - binary_crossentropy: 0.6454 - auc: 0.6772 - precision: 0.6260 - recall: 0.6390\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6454 - binary_accuracy: 0.6309 - binary_crossentropy: 0.6454 - auc: 0.6772 - precision: 0.6260 - recall: 0.6391\n",
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6455 - binary_accuracy: 0.6307 - binary_crossentropy: 0.6455 - auc: 0.6771 - precision: 0.6258 - recall: 0.6390\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6454 - binary_accuracy: 0.6307 - binary_crossentropy: 0.6454 - auc: 0.6771 - precision: 0.6258 - recall: 0.6390\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6454 - binary_accuracy: 0.6308 - binary_crossentropy: 0.6454 - auc: 0.6772 - precision: 0.6258 - recall: 0.6390\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 783ms/step - loss: 0.6454 - binary_accuracy: 0.6308 - binary_crossentropy: 0.6454 - auc: 0.6772 - precision: 0.6258 - recall: 0.6390 - val_loss: 0.6314 - val_binary_accuracy: 0.6533 - val_binary_crossentropy: 0.6314 - val_auc: 0.7011 - val_precision: 0.6640 - val_recall: 0.6383\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 13/585\n",
      " 1/50 [..............................] - ETA: 10s - loss: 0.6440 - binary_accuracy: 0.6320 - binary_crossentropy: 0.6440 - auc: 0.6793 - precision: 0.6309 - recall: 0.6373\n",
      " 2/50 [>.............................] - ETA: 10s - loss: 0.6436 - binary_accuracy: 0.6321 - binary_crossentropy: 0.6436 - auc: 0.6794 - precision: 0.6287 - recall: 0.6397\n",
      " 3/50 [>.............................] - ETA: 9s - loss: 0.6448 - binary_accuracy: 0.6309 - binary_crossentropy: 0.6448 - auc: 0.6778 - precision: 0.6265 - recall: 0.6404 \n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.6444 - binary_accuracy: 0.6317 - binary_crossentropy: 0.6444 - auc: 0.6788 - precision: 0.6261 - recall: 0.6413\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6448 - binary_accuracy: 0.6313 - binary_crossentropy: 0.6448 - auc: 0.6784 - precision: 0.6256 - recall: 0.6415\n",
      " 6/50 [==>...........................] - ETA: 8s - loss: 0.6449 - binary_accuracy: 0.6310 - binary_crossentropy: 0.6449 - auc: 0.6781 - precision: 0.6249 - recall: 0.6412\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6448 - binary_accuracy: 0.6310 - binary_crossentropy: 0.6448 - auc: 0.6783 - precision: 0.6248 - recall: 0.6410\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6449 - binary_accuracy: 0.6308 - binary_crossentropy: 0.6449 - auc: 0.6781 - precision: 0.6244 - recall: 0.6403\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6449 - binary_accuracy: 0.6307 - binary_crossentropy: 0.6449 - auc: 0.6778 - precision: 0.6250 - recall: 0.6394\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6446 - binary_accuracy: 0.6313 - binary_crossentropy: 0.6446 - auc: 0.6782 - precision: 0.6256 - recall: 0.6397\n",
      "11/50 [=====>........................] - ETA: 6s - loss: 0.6444 - binary_accuracy: 0.6312 - binary_crossentropy: 0.6444 - auc: 0.6784 - precision: 0.6256 - recall: 0.6392\n",
      "12/50 [======>.......................] - ETA: 6s - loss: 0.6443 - binary_accuracy: 0.6317 - binary_crossentropy: 0.6443 - auc: 0.6787 - precision: 0.6263 - recall: 0.6393\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6445 - binary_accuracy: 0.6312 - binary_crossentropy: 0.6445 - auc: 0.6784 - precision: 0.6258 - recall: 0.6388\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6447 - binary_accuracy: 0.6312 - binary_crossentropy: 0.6447 - auc: 0.6783 - precision: 0.6260 - recall: 0.6384\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6449 - binary_accuracy: 0.6310 - binary_crossentropy: 0.6449 - auc: 0.6780 - precision: 0.6258 - recall: 0.6382\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6451 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6451 - auc: 0.6777 - precision: 0.6255 - recall: 0.6378\n",
      "17/50 [=========>....................] - ETA: 5s - loss: 0.6451 - binary_accuracy: 0.6307 - binary_crossentropy: 0.6451 - auc: 0.6777 - precision: 0.6255 - recall: 0.6378\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6451 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6451 - auc: 0.6775 - precision: 0.6257 - recall: 0.6377\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6451 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6451 - auc: 0.6775 - precision: 0.6259 - recall: 0.6374\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6450 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6450 - auc: 0.6776 - precision: 0.6254 - recall: 0.6380\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6450 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6450 - auc: 0.6775 - precision: 0.6255 - recall: 0.6385\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6450 - binary_accuracy: 0.6304 - binary_crossentropy: 0.6450 - auc: 0.6775 - precision: 0.6251 - recall: 0.6386\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6450 - binary_accuracy: 0.6304 - binary_crossentropy: 0.6450 - auc: 0.6776 - precision: 0.6251 - recall: 0.6388\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6449 - binary_accuracy: 0.6304 - binary_crossentropy: 0.6449 - auc: 0.6777 - precision: 0.6249 - recall: 0.6391\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6449 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6449 - auc: 0.6777 - precision: 0.6251 - recall: 0.6390\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6450 - binary_accuracy: 0.6304 - binary_crossentropy: 0.6450 - auc: 0.6777 - precision: 0.6250 - recall: 0.6387\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6449 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6449 - auc: 0.6777 - precision: 0.6250 - recall: 0.6387\n",
      "28/50 [===============>..............] - ETA: 3s - loss: 0.6450 - binary_accuracy: 0.6304 - binary_crossentropy: 0.6450 - auc: 0.6776 - precision: 0.6248 - recall: 0.6386\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6451 - binary_accuracy: 0.6303 - binary_crossentropy: 0.6451 - auc: 0.6774 - precision: 0.6249 - recall: 0.6383\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6452 - binary_accuracy: 0.6302 - binary_crossentropy: 0.6452 - auc: 0.6773 - precision: 0.6250 - recall: 0.6380\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6452 - binary_accuracy: 0.6303 - binary_crossentropy: 0.6452 - auc: 0.6773 - precision: 0.6252 - recall: 0.6380\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6451 - binary_accuracy: 0.6304 - binary_crossentropy: 0.6451 - auc: 0.6774 - precision: 0.6253 - recall: 0.6379\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6452 - binary_accuracy: 0.6304 - binary_crossentropy: 0.6452 - auc: 0.6773 - precision: 0.6255 - recall: 0.6376\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6452 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6452 - auc: 0.6773 - precision: 0.6256 - recall: 0.6376\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6453 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6453 - auc: 0.6773 - precision: 0.6256 - recall: 0.6375\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6452 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6452 - auc: 0.6774 - precision: 0.6258 - recall: 0.6378\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6452 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6452 - auc: 0.6773 - precision: 0.6258 - recall: 0.6377\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6452 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6452 - auc: 0.6772 - precision: 0.6256 - recall: 0.6379\n",
      "39/50 [======================>.......] - ETA: 1s - loss: 0.6451 - binary_accuracy: 0.6307 - binary_crossentropy: 0.6451 - auc: 0.6774 - precision: 0.6257 - recall: 0.6384\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6451 - binary_accuracy: 0.6307 - binary_crossentropy: 0.6451 - auc: 0.6774 - precision: 0.6257 - recall: 0.6386\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6452 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6452 - auc: 0.6773 - precision: 0.6257 - recall: 0.6388\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6452 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6452 - auc: 0.6772 - precision: 0.6256 - recall: 0.6387\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6453 - binary_accuracy: 0.6305 - binary_crossentropy: 0.6453 - auc: 0.6772 - precision: 0.6255 - recall: 0.6388\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6452 - binary_accuracy: 0.6306 - binary_crossentropy: 0.6452 - auc: 0.6772 - precision: 0.6257 - recall: 0.6388\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6452 - binary_accuracy: 0.6307 - binary_crossentropy: 0.6452 - auc: 0.6773 - precision: 0.6260 - recall: 0.6387\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6452 - binary_accuracy: 0.6308 - binary_crossentropy: 0.6452 - auc: 0.6774 - precision: 0.6260 - recall: 0.6389\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6452 - binary_accuracy: 0.6308 - binary_crossentropy: 0.6452 - auc: 0.6774 - precision: 0.6259 - recall: 0.6391\n",
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6452 - binary_accuracy: 0.6308 - binary_crossentropy: 0.6452 - auc: 0.6774 - precision: 0.6258 - recall: 0.6391\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6452 - binary_accuracy: 0.6308 - binary_crossentropy: 0.6452 - auc: 0.6774 - precision: 0.6258 - recall: 0.6392\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6451 - binary_accuracy: 0.6309 - binary_crossentropy: 0.6451 - auc: 0.6774 - precision: 0.6259 - recall: 0.6394\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 785ms/step - loss: 0.6451 - binary_accuracy: 0.6309 - binary_crossentropy: 0.6451 - auc: 0.6774 - precision: 0.6259 - recall: 0.6394 - val_loss: 0.6290 - val_binary_accuracy: 0.6559 - val_binary_crossentropy: 0.6290 - val_auc: 0.7029 - val_precision: 0.6591 - val_recall: 0.6639\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 14/585\n",
      " 1/50 [..............................] - ETA: 12s - loss: 0.6470 - binary_accuracy: 0.6288 - binary_crossentropy: 0.6470 - auc: 0.6754 - precision: 0.6250 - recall: 0.6384\n",
      " 2/50 [>.............................] - ETA: 8s - loss: 0.6477 - binary_accuracy: 0.6288 - binary_crossentropy: 0.6477 - auc: 0.6737 - precision: 0.6227 - recall: 0.6397 \n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6470 - binary_accuracy: 0.6284 - binary_crossentropy: 0.6470 - auc: 0.6746 - precision: 0.6218 - recall: 0.6393\n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.6453 - binary_accuracy: 0.6307 - binary_crossentropy: 0.6453 - auc: 0.6770 - precision: 0.6237 - recall: 0.6408\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6447 - binary_accuracy: 0.6308 - binary_crossentropy: 0.6447 - auc: 0.6777 - precision: 0.6250 - recall: 0.6402\n",
      " 6/50 [==>...........................] - ETA: 7s - loss: 0.6444 - binary_accuracy: 0.6310 - binary_crossentropy: 0.6444 - auc: 0.6781 - precision: 0.6260 - recall: 0.6393\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6445 - binary_accuracy: 0.6309 - binary_crossentropy: 0.6445 - auc: 0.6780 - precision: 0.6265 - recall: 0.6384\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6444 - binary_accuracy: 0.6313 - binary_crossentropy: 0.6444 - auc: 0.6784 - precision: 0.6272 - recall: 0.6388\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6447 - binary_accuracy: 0.6311 - binary_crossentropy: 0.6447 - auc: 0.6781 - precision: 0.6273 - recall: 0.6384\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6448 - binary_accuracy: 0.6312 - binary_crossentropy: 0.6448 - auc: 0.6780 - precision: 0.6272 - recall: 0.6389\n",
      "11/50 [=====>........................] - ETA: 7s - loss: 0.6448 - binary_accuracy: 0.6316 - binary_crossentropy: 0.6448 - auc: 0.6782 - precision: 0.6281 - recall: 0.6400\n",
      "12/50 [======>.......................] - ETA: 6s - loss: 0.6447 - binary_accuracy: 0.6318 - binary_crossentropy: 0.6447 - auc: 0.6782 - precision: 0.6286 - recall: 0.6405\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6446 - binary_accuracy: 0.6318 - binary_crossentropy: 0.6446 - auc: 0.6783 - precision: 0.6289 - recall: 0.6410\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6447 - binary_accuracy: 0.6316 - binary_crossentropy: 0.6447 - auc: 0.6780 - precision: 0.6287 - recall: 0.6414\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6448 - binary_accuracy: 0.6315 - binary_crossentropy: 0.6448 - auc: 0.6779 - precision: 0.6283 - recall: 0.6417\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6449 - binary_accuracy: 0.6313 - binary_crossentropy: 0.6449 - auc: 0.6777 - precision: 0.6279 - recall: 0.6421\n",
      "17/50 [=========>....................] - ETA: 6s - loss: 0.6447 - binary_accuracy: 0.6313 - binary_crossentropy: 0.6447 - auc: 0.6779 - precision: 0.6274 - recall: 0.6430\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6446 - binary_accuracy: 0.6312 - binary_crossentropy: 0.6446 - auc: 0.6781 - precision: 0.6271 - recall: 0.6433\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6446 - binary_accuracy: 0.6313 - binary_crossentropy: 0.6446 - auc: 0.6781 - precision: 0.6270 - recall: 0.6438\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6445 - binary_accuracy: 0.6312 - binary_crossentropy: 0.6445 - auc: 0.6782 - precision: 0.6269 - recall: 0.6442\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6448 - binary_accuracy: 0.6308 - binary_crossentropy: 0.6448 - auc: 0.6777 - precision: 0.6265 - recall: 0.6441\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6447 - binary_accuracy: 0.6309 - binary_crossentropy: 0.6447 - auc: 0.6778 - precision: 0.6264 - recall: 0.6444\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6447 - binary_accuracy: 0.6311 - binary_crossentropy: 0.6447 - auc: 0.6780 - precision: 0.6265 - recall: 0.6446\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6446 - binary_accuracy: 0.6313 - binary_crossentropy: 0.6446 - auc: 0.6781 - precision: 0.6267 - recall: 0.6450\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6445 - binary_accuracy: 0.6315 - binary_crossentropy: 0.6445 - auc: 0.6782 - precision: 0.6268 - recall: 0.6451\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6445 - binary_accuracy: 0.6316 - binary_crossentropy: 0.6445 - auc: 0.6783 - precision: 0.6272 - recall: 0.6449\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6445 - binary_accuracy: 0.6315 - binary_crossentropy: 0.6445 - auc: 0.6782 - precision: 0.6270 - recall: 0.6446\n",
      "28/50 [===============>..............] - ETA: 4s - loss: 0.6446 - binary_accuracy: 0.6315 - binary_crossentropy: 0.6446 - auc: 0.6781 - precision: 0.6268 - recall: 0.6445\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6446 - binary_accuracy: 0.6314 - binary_crossentropy: 0.6446 - auc: 0.6781 - precision: 0.6268 - recall: 0.6441\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6447 - binary_accuracy: 0.6314 - binary_crossentropy: 0.6447 - auc: 0.6780 - precision: 0.6267 - recall: 0.6439\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6446 - binary_accuracy: 0.6314 - binary_crossentropy: 0.6446 - auc: 0.6781 - precision: 0.6268 - recall: 0.6436\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6445 - binary_accuracy: 0.6315 - binary_crossentropy: 0.6445 - auc: 0.6782 - precision: 0.6270 - recall: 0.6434\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6446 - binary_accuracy: 0.6314 - binary_crossentropy: 0.6446 - auc: 0.6782 - precision: 0.6268 - recall: 0.6431\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6446 - binary_accuracy: 0.6314 - binary_crossentropy: 0.6446 - auc: 0.6781 - precision: 0.6267 - recall: 0.6429\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6446 - binary_accuracy: 0.6314 - binary_crossentropy: 0.6446 - auc: 0.6781 - precision: 0.6267 - recall: 0.6427\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6448 - binary_accuracy: 0.6311 - binary_crossentropy: 0.6448 - auc: 0.6778 - precision: 0.6266 - recall: 0.6423\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6448 - binary_accuracy: 0.6312 - binary_crossentropy: 0.6448 - auc: 0.6778 - precision: 0.6266 - recall: 0.6422\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6447 - binary_accuracy: 0.6312 - binary_crossentropy: 0.6447 - auc: 0.6779 - precision: 0.6267 - recall: 0.6424\n",
      "39/50 [======================>.......] - ETA: 2s - loss: 0.6448 - binary_accuracy: 0.6312 - binary_crossentropy: 0.6448 - auc: 0.6778 - precision: 0.6266 - recall: 0.6424\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6448 - binary_accuracy: 0.6312 - binary_crossentropy: 0.6448 - auc: 0.6778 - precision: 0.6265 - recall: 0.6424\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6448 - binary_accuracy: 0.6312 - binary_crossentropy: 0.6448 - auc: 0.6778 - precision: 0.6266 - recall: 0.6425\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6448 - binary_accuracy: 0.6313 - binary_crossentropy: 0.6448 - auc: 0.6779 - precision: 0.6266 - recall: 0.6426\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6447 - binary_accuracy: 0.6314 - binary_crossentropy: 0.6447 - auc: 0.6780 - precision: 0.6267 - recall: 0.6428\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6447 - binary_accuracy: 0.6314 - binary_crossentropy: 0.6447 - auc: 0.6780 - precision: 0.6268 - recall: 0.6429\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6446 - binary_accuracy: 0.6316 - binary_crossentropy: 0.6446 - auc: 0.6781 - precision: 0.6270 - recall: 0.6430\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6446 - binary_accuracy: 0.6316 - binary_crossentropy: 0.6446 - auc: 0.6781 - precision: 0.6271 - recall: 0.6431\n",
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6446 - binary_accuracy: 0.6317 - binary_crossentropy: 0.6446 - auc: 0.6781 - precision: 0.6272 - recall: 0.6431\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6445 - binary_accuracy: 0.6318 - binary_crossentropy: 0.6445 - auc: 0.6782 - precision: 0.6271 - recall: 0.6434\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6445 - binary_accuracy: 0.6318 - binary_crossentropy: 0.6445 - auc: 0.6782 - precision: 0.6270 - recall: 0.6435\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6445 - binary_accuracy: 0.6318 - binary_crossentropy: 0.6445 - auc: 0.6782 - precision: 0.6270 - recall: 0.6436\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 794ms/step - loss: 0.6445 - binary_accuracy: 0.6318 - binary_crossentropy: 0.6445 - auc: 0.6782 - precision: 0.6270 - recall: 0.6436 - val_loss: 0.6288 - val_binary_accuracy: 0.6549 - val_binary_crossentropy: 0.6288 - val_auc: 0.7026 - val_precision: 0.6555 - val_recall: 0.6713\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 15/585\n",
      " 1/50 [..............................] - ETA: 9s - loss: 0.6448 - binary_accuracy: 0.6301 - binary_crossentropy: 0.6448 - auc: 0.6785 - precision: 0.6212 - recall: 0.6437\n",
      " 2/50 [>.............................] - ETA: 9s - loss: 0.6427 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6427 - auc: 0.6814 - precision: 0.6240 - recall: 0.6467\n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6430 - binary_accuracy: 0.6317 - binary_crossentropy: 0.6430 - auc: 0.6807 - precision: 0.6256 - recall: 0.6447\n",
      " 4/50 [=>............................] - ETA: 9s - loss: 0.6437 - binary_accuracy: 0.6315 - binary_crossentropy: 0.6437 - auc: 0.6797 - precision: 0.6264 - recall: 0.6442\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6437 - binary_accuracy: 0.6321 - binary_crossentropy: 0.6437 - auc: 0.6799 - precision: 0.6277 - recall: 0.6429\n",
      " 6/50 [==>...........................] - ETA: 8s - loss: 0.6436 - binary_accuracy: 0.6324 - binary_crossentropy: 0.6436 - auc: 0.6798 - precision: 0.6280 - recall: 0.6423\n",
      " 7/50 [===>..........................] - ETA: 8s - loss: 0.6437 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6437 - auc: 0.6796 - precision: 0.6279 - recall: 0.6417\n",
      " 8/50 [===>..........................] - ETA: 8s - loss: 0.6438 - binary_accuracy: 0.6324 - binary_crossentropy: 0.6438 - auc: 0.6794 - precision: 0.6276 - recall: 0.6411\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6439 - binary_accuracy: 0.6321 - binary_crossentropy: 0.6439 - auc: 0.6793 - precision: 0.6273 - recall: 0.6406\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6435 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6435 - auc: 0.6799 - precision: 0.6280 - recall: 0.6410\n",
      "11/50 [=====>........................] - ETA: 7s - loss: 0.6434 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6434 - auc: 0.6801 - precision: 0.6285 - recall: 0.6416\n",
      "12/50 [======>.......................] - ETA: 7s - loss: 0.6435 - binary_accuracy: 0.6332 - binary_crossentropy: 0.6435 - auc: 0.6801 - precision: 0.6289 - recall: 0.6418\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6437 - binary_accuracy: 0.6328 - binary_crossentropy: 0.6437 - auc: 0.6797 - precision: 0.6286 - recall: 0.6413\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6438 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6438 - auc: 0.6796 - precision: 0.6289 - recall: 0.6411\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6438 - binary_accuracy: 0.6325 - binary_crossentropy: 0.6438 - auc: 0.6795 - precision: 0.6291 - recall: 0.6411\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6438 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6438 - auc: 0.6796 - precision: 0.6290 - recall: 0.6416\n",
      "17/50 [=========>....................] - ETA: 6s - loss: 0.6437 - binary_accuracy: 0.6328 - binary_crossentropy: 0.6437 - auc: 0.6797 - precision: 0.6291 - recall: 0.6420\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6436 - binary_accuracy: 0.6328 - binary_crossentropy: 0.6436 - auc: 0.6797 - precision: 0.6289 - recall: 0.6423\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6437 - binary_accuracy: 0.6328 - binary_crossentropy: 0.6437 - auc: 0.6796 - precision: 0.6285 - recall: 0.6429\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6437 - binary_accuracy: 0.6329 - binary_crossentropy: 0.6437 - auc: 0.6796 - precision: 0.6285 - recall: 0.6436\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6437 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6437 - auc: 0.6796 - precision: 0.6283 - recall: 0.6441\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6437 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6437 - auc: 0.6796 - precision: 0.6281 - recall: 0.6445\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6437 - binary_accuracy: 0.6331 - binary_crossentropy: 0.6437 - auc: 0.6796 - precision: 0.6281 - recall: 0.6447\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6439 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6439 - auc: 0.6795 - precision: 0.6281 - recall: 0.6447\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6439 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6439 - auc: 0.6793 - precision: 0.6280 - recall: 0.6447\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6438 - binary_accuracy: 0.6331 - binary_crossentropy: 0.6438 - auc: 0.6795 - precision: 0.6283 - recall: 0.6446\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6438 - binary_accuracy: 0.6331 - binary_crossentropy: 0.6438 - auc: 0.6795 - precision: 0.6284 - recall: 0.6443\n",
      "28/50 [===============>..............] - ETA: 3s - loss: 0.6439 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6439 - auc: 0.6793 - precision: 0.6283 - recall: 0.6439\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6440 - binary_accuracy: 0.6328 - binary_crossentropy: 0.6440 - auc: 0.6791 - precision: 0.6283 - recall: 0.6435\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6440 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6440 - auc: 0.6791 - precision: 0.6282 - recall: 0.6435\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6439 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6439 - auc: 0.6791 - precision: 0.6278 - recall: 0.6435\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6439 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6439 - auc: 0.6791 - precision: 0.6277 - recall: 0.6435\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6440 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6440 - auc: 0.6791 - precision: 0.6276 - recall: 0.6433\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6440 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6440 - auc: 0.6791 - precision: 0.6275 - recall: 0.6433\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6440 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6440 - auc: 0.6791 - precision: 0.6275 - recall: 0.6431\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6439 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6439 - auc: 0.6791 - precision: 0.6275 - recall: 0.6428\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6439 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6439 - auc: 0.6791 - precision: 0.6275 - recall: 0.6427\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6440 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6440 - auc: 0.6790 - precision: 0.6276 - recall: 0.6424\n",
      "39/50 [======================>.......] - ETA: 1s - loss: 0.6440 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6440 - auc: 0.6791 - precision: 0.6277 - recall: 0.6423\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6440 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6440 - auc: 0.6791 - precision: 0.6276 - recall: 0.6421\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6439 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6439 - auc: 0.6791 - precision: 0.6276 - recall: 0.6421\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6440 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6440 - auc: 0.6791 - precision: 0.6276 - recall: 0.6421\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6439 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6439 - auc: 0.6791 - precision: 0.6277 - recall: 0.6422\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6439 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6439 - auc: 0.6792 - precision: 0.6278 - recall: 0.6422\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6439 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6439 - auc: 0.6792 - precision: 0.6277 - recall: 0.6422\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6439 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6439 - auc: 0.6791 - precision: 0.6276 - recall: 0.6421\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6439 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6439 - auc: 0.6791 - precision: 0.6276 - recall: 0.6421\n",
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6440 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6440 - auc: 0.6791 - precision: 0.6276 - recall: 0.6420\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6439 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6439 - auc: 0.6792 - precision: 0.6277 - recall: 0.6420\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6439 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6439 - auc: 0.6792 - precision: 0.6277 - recall: 0.6420\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 789ms/step - loss: 0.6439 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6439 - auc: 0.6792 - precision: 0.6277 - recall: 0.6420 - val_loss: 0.6287 - val_binary_accuracy: 0.6560 - val_binary_crossentropy: 0.6287 - val_auc: 0.7037 - val_precision: 0.6638 - val_recall: 0.6496\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 16/585\n",
      " 1/50 [..............................] - ETA: 10s - loss: 0.6453 - binary_accuracy: 0.6283 - binary_crossentropy: 0.6453 - auc: 0.6764 - precision: 0.6227 - recall: 0.6390\n",
      " 2/50 [>.............................] - ETA: 10s - loss: 0.6464 - binary_accuracy: 0.6277 - binary_crossentropy: 0.6464 - auc: 0.6750 - precision: 0.6247 - recall: 0.6378\n",
      " 3/50 [>.............................] - ETA: 9s - loss: 0.6453 - binary_accuracy: 0.6294 - binary_crossentropy: 0.6453 - auc: 0.6768 - precision: 0.6273 - recall: 0.6415 \n",
      " 4/50 [=>............................] - ETA: 9s - loss: 0.6456 - binary_accuracy: 0.6294 - binary_crossentropy: 0.6456 - auc: 0.6766 - precision: 0.6263 - recall: 0.6417\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6455 - binary_accuracy: 0.6303 - binary_crossentropy: 0.6455 - auc: 0.6767 - precision: 0.6254 - recall: 0.6434\n",
      " 6/50 [==>...........................] - ETA: 8s - loss: 0.6451 - binary_accuracy: 0.6307 - binary_crossentropy: 0.6451 - auc: 0.6772 - precision: 0.6263 - recall: 0.6435\n",
      " 7/50 [===>..........................] - ETA: 8s - loss: 0.6447 - binary_accuracy: 0.6317 - binary_crossentropy: 0.6447 - auc: 0.6778 - precision: 0.6273 - recall: 0.6448\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6446 - binary_accuracy: 0.6319 - binary_crossentropy: 0.6446 - auc: 0.6779 - precision: 0.6276 - recall: 0.6450\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6442 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6442 - auc: 0.6786 - precision: 0.6281 - recall: 0.6460\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6439 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6439 - auc: 0.6789 - precision: 0.6285 - recall: 0.6461\n",
      "11/50 [=====>........................] - ETA: 7s - loss: 0.6436 - binary_accuracy: 0.6334 - binary_crossentropy: 0.6436 - auc: 0.6794 - precision: 0.6289 - recall: 0.6465\n",
      "12/50 [======>.......................] - ETA: 6s - loss: 0.6433 - binary_accuracy: 0.6336 - binary_crossentropy: 0.6433 - auc: 0.6798 - precision: 0.6290 - recall: 0.6468\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6435 - binary_accuracy: 0.6333 - binary_crossentropy: 0.6435 - auc: 0.6796 - precision: 0.6286 - recall: 0.6467\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6434 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6434 - auc: 0.6797 - precision: 0.6289 - recall: 0.6471\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6436 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6436 - auc: 0.6793 - precision: 0.6283 - recall: 0.6467\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6437 - binary_accuracy: 0.6328 - binary_crossentropy: 0.6437 - auc: 0.6792 - precision: 0.6282 - recall: 0.6464\n",
      "17/50 [=========>....................] - ETA: 6s - loss: 0.6439 - binary_accuracy: 0.6323 - binary_crossentropy: 0.6439 - auc: 0.6789 - precision: 0.6276 - recall: 0.6457\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6439 - binary_accuracy: 0.6323 - binary_crossentropy: 0.6439 - auc: 0.6789 - precision: 0.6276 - recall: 0.6457\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6439 - binary_accuracy: 0.6321 - binary_crossentropy: 0.6439 - auc: 0.6787 - precision: 0.6272 - recall: 0.6455\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6438 - binary_accuracy: 0.6324 - binary_crossentropy: 0.6438 - auc: 0.6790 - precision: 0.6273 - recall: 0.6457\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6436 - binary_accuracy: 0.6325 - binary_crossentropy: 0.6436 - auc: 0.6792 - precision: 0.6278 - recall: 0.6455\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6437 - binary_accuracy: 0.6323 - binary_crossentropy: 0.6437 - auc: 0.6791 - precision: 0.6274 - recall: 0.6452\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6438 - binary_accuracy: 0.6322 - binary_crossentropy: 0.6438 - auc: 0.6790 - precision: 0.6274 - recall: 0.6449\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6439 - binary_accuracy: 0.6320 - binary_crossentropy: 0.6439 - auc: 0.6788 - precision: 0.6274 - recall: 0.6443\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6438 - binary_accuracy: 0.6322 - binary_crossentropy: 0.6438 - auc: 0.6790 - precision: 0.6278 - recall: 0.6442\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6438 - binary_accuracy: 0.6321 - binary_crossentropy: 0.6438 - auc: 0.6789 - precision: 0.6278 - recall: 0.6441\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6439 - binary_accuracy: 0.6321 - binary_crossentropy: 0.6439 - auc: 0.6788 - precision: 0.6277 - recall: 0.6442\n",
      "28/50 [===============>..............] - ETA: 4s - loss: 0.6439 - binary_accuracy: 0.6321 - binary_crossentropy: 0.6439 - auc: 0.6788 - precision: 0.6276 - recall: 0.6443\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6440 - binary_accuracy: 0.6320 - binary_crossentropy: 0.6440 - auc: 0.6786 - precision: 0.6275 - recall: 0.6442\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6440 - binary_accuracy: 0.6321 - binary_crossentropy: 0.6440 - auc: 0.6786 - precision: 0.6275 - recall: 0.6443\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6438 - binary_accuracy: 0.6323 - binary_crossentropy: 0.6438 - auc: 0.6788 - precision: 0.6277 - recall: 0.6445\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6438 - binary_accuracy: 0.6323 - binary_crossentropy: 0.6438 - auc: 0.6789 - precision: 0.6278 - recall: 0.6445\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6437 - binary_accuracy: 0.6324 - binary_crossentropy: 0.6437 - auc: 0.6789 - precision: 0.6276 - recall: 0.6447\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6437 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6437 - auc: 0.6790 - precision: 0.6277 - recall: 0.6450\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6436 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6436 - auc: 0.6792 - precision: 0.6279 - recall: 0.6451\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6435 - binary_accuracy: 0.6329 - binary_crossentropy: 0.6435 - auc: 0.6793 - precision: 0.6280 - recall: 0.6452\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6434 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6434 - auc: 0.6794 - precision: 0.6281 - recall: 0.6453\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6434 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6434 - auc: 0.6795 - precision: 0.6281 - recall: 0.6451\n",
      "39/50 [======================>.......] - ETA: 2s - loss: 0.6435 - binary_accuracy: 0.6329 - binary_crossentropy: 0.6435 - auc: 0.6794 - precision: 0.6280 - recall: 0.6450\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6435 - binary_accuracy: 0.6329 - binary_crossentropy: 0.6435 - auc: 0.6794 - precision: 0.6279 - recall: 0.6449\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6434 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6434 - auc: 0.6796 - precision: 0.6281 - recall: 0.6450\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6434 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6434 - auc: 0.6796 - precision: 0.6282 - recall: 0.6449\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6433 - binary_accuracy: 0.6330 - binary_crossentropy: 0.6433 - auc: 0.6797 - precision: 0.6281 - recall: 0.6448\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6433 - binary_accuracy: 0.6331 - binary_crossentropy: 0.6433 - auc: 0.6797 - precision: 0.6283 - recall: 0.6448\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6433 - binary_accuracy: 0.6331 - binary_crossentropy: 0.6433 - auc: 0.6797 - precision: 0.6283 - recall: 0.6447\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6432 - binary_accuracy: 0.6332 - binary_crossentropy: 0.6432 - auc: 0.6799 - precision: 0.6284 - recall: 0.6448\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6431 - binary_accuracy: 0.6333 - binary_crossentropy: 0.6431 - auc: 0.6799 - precision: 0.6286 - recall: 0.6448\n",
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6431 - binary_accuracy: 0.6333 - binary_crossentropy: 0.6431 - auc: 0.6800 - precision: 0.6285 - recall: 0.6448\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6431 - binary_accuracy: 0.6334 - binary_crossentropy: 0.6431 - auc: 0.6801 - precision: 0.6285 - recall: 0.6449\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6430 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6430 - auc: 0.6801 - precision: 0.6286 - recall: 0.6450\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 792ms/step - loss: 0.6430 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6430 - auc: 0.6801 - precision: 0.6286 - recall: 0.6450 - val_loss: 0.6292 - val_binary_accuracy: 0.6555 - val_binary_crossentropy: 0.6292 - val_auc: 0.7046 - val_precision: 0.6697 - val_recall: 0.6308\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 17/585\n",
      " 1/50 [..............................] - ETA: 11s - loss: 0.6447 - binary_accuracy: 0.6342 - binary_crossentropy: 0.6447 - auc: 0.6785 - precision: 0.6285 - recall: 0.6445\n",
      " 2/50 [>.............................] - ETA: 9s - loss: 0.6431 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6431 - auc: 0.6807 - precision: 0.6294 - recall: 0.6462 \n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6424 - binary_accuracy: 0.6361 - binary_crossentropy: 0.6424 - auc: 0.6817 - precision: 0.6311 - recall: 0.6476\n",
      " 4/50 [=>............................] - ETA: 9s - loss: 0.6434 - binary_accuracy: 0.6341 - binary_crossentropy: 0.6434 - auc: 0.6799 - precision: 0.6286 - recall: 0.6460\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6430 - binary_accuracy: 0.6337 - binary_crossentropy: 0.6430 - auc: 0.6805 - precision: 0.6283 - recall: 0.6456\n",
      " 6/50 [==>...........................] - ETA: 8s - loss: 0.6428 - binary_accuracy: 0.6336 - binary_crossentropy: 0.6428 - auc: 0.6806 - precision: 0.6290 - recall: 0.6447\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6424 - binary_accuracy: 0.6340 - binary_crossentropy: 0.6424 - auc: 0.6812 - precision: 0.6295 - recall: 0.6447\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6429 - binary_accuracy: 0.6334 - binary_crossentropy: 0.6429 - auc: 0.6804 - precision: 0.6289 - recall: 0.6440\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6431 - binary_accuracy: 0.6334 - binary_crossentropy: 0.6431 - auc: 0.6803 - precision: 0.6290 - recall: 0.6440\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6438 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6438 - auc: 0.6792 - precision: 0.6285 - recall: 0.6432\n",
      "11/50 [=====>........................] - ETA: 6s - loss: 0.6441 - binary_accuracy: 0.6324 - binary_crossentropy: 0.6441 - auc: 0.6786 - precision: 0.6284 - recall: 0.6429\n",
      "12/50 [======>.......................] - ETA: 6s - loss: 0.6439 - binary_accuracy: 0.6327 - binary_crossentropy: 0.6439 - auc: 0.6790 - precision: 0.6288 - recall: 0.6434\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6440 - binary_accuracy: 0.6324 - binary_crossentropy: 0.6440 - auc: 0.6788 - precision: 0.6279 - recall: 0.6434\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6439 - binary_accuracy: 0.6324 - binary_crossentropy: 0.6439 - auc: 0.6789 - precision: 0.6277 - recall: 0.6436\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6439 - binary_accuracy: 0.6323 - binary_crossentropy: 0.6439 - auc: 0.6789 - precision: 0.6278 - recall: 0.6436\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6437 - binary_accuracy: 0.6325 - binary_crossentropy: 0.6437 - auc: 0.6791 - precision: 0.6277 - recall: 0.6441\n",
      "17/50 [=========>....................] - ETA: 5s - loss: 0.6438 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6438 - auc: 0.6790 - precision: 0.6274 - recall: 0.6444\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6438 - binary_accuracy: 0.6325 - binary_crossentropy: 0.6438 - auc: 0.6789 - precision: 0.6275 - recall: 0.6443\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6438 - binary_accuracy: 0.6325 - binary_crossentropy: 0.6438 - auc: 0.6789 - precision: 0.6276 - recall: 0.6442\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6438 - binary_accuracy: 0.6325 - binary_crossentropy: 0.6438 - auc: 0.6791 - precision: 0.6278 - recall: 0.6442\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6437 - binary_accuracy: 0.6326 - binary_crossentropy: 0.6437 - auc: 0.6792 - precision: 0.6278 - recall: 0.6441\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6435 - binary_accuracy: 0.6329 - binary_crossentropy: 0.6435 - auc: 0.6796 - precision: 0.6284 - recall: 0.6444\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6434 - binary_accuracy: 0.6332 - binary_crossentropy: 0.6434 - auc: 0.6797 - precision: 0.6287 - recall: 0.6446\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6433 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6433 - auc: 0.6799 - precision: 0.6289 - recall: 0.6449\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6433 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6433 - auc: 0.6798 - precision: 0.6289 - recall: 0.6449\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6433 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6433 - auc: 0.6798 - precision: 0.6287 - recall: 0.6451\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6433 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6433 - auc: 0.6798 - precision: 0.6286 - recall: 0.6453\n",
      "28/50 [===============>..............] - ETA: 3s - loss: 0.6433 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6433 - auc: 0.6799 - precision: 0.6285 - recall: 0.6454\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6432 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6432 - auc: 0.6799 - precision: 0.6286 - recall: 0.6453\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6432 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6432 - auc: 0.6799 - precision: 0.6287 - recall: 0.6453\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6432 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6432 - auc: 0.6799 - precision: 0.6288 - recall: 0.6450\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6432 - binary_accuracy: 0.6336 - binary_crossentropy: 0.6432 - auc: 0.6799 - precision: 0.6288 - recall: 0.6452\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6431 - binary_accuracy: 0.6336 - binary_crossentropy: 0.6431 - auc: 0.6801 - precision: 0.6289 - recall: 0.6451\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6432 - binary_accuracy: 0.6336 - binary_crossentropy: 0.6432 - auc: 0.6801 - precision: 0.6289 - recall: 0.6453\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6432 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6432 - auc: 0.6799 - precision: 0.6287 - recall: 0.6450\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6432 - binary_accuracy: 0.6336 - binary_crossentropy: 0.6432 - auc: 0.6800 - precision: 0.6288 - recall: 0.6450\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6432 - binary_accuracy: 0.6336 - binary_crossentropy: 0.6432 - auc: 0.6800 - precision: 0.6287 - recall: 0.6448\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6433 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6433 - auc: 0.6799 - precision: 0.6287 - recall: 0.6446\n",
      "39/50 [======================>.......] - ETA: 1s - loss: 0.6433 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6433 - auc: 0.6798 - precision: 0.6288 - recall: 0.6444\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6434 - binary_accuracy: 0.6334 - binary_crossentropy: 0.6434 - auc: 0.6797 - precision: 0.6287 - recall: 0.6442\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6433 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6433 - auc: 0.6798 - precision: 0.6288 - recall: 0.6442\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6434 - binary_accuracy: 0.6334 - binary_crossentropy: 0.6434 - auc: 0.6797 - precision: 0.6286 - recall: 0.6443\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6434 - binary_accuracy: 0.6334 - binary_crossentropy: 0.6434 - auc: 0.6797 - precision: 0.6287 - recall: 0.6442\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6433 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6433 - auc: 0.6798 - precision: 0.6287 - recall: 0.6444\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6432 - binary_accuracy: 0.6336 - binary_crossentropy: 0.6432 - auc: 0.6799 - precision: 0.6289 - recall: 0.6444\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6432 - binary_accuracy: 0.6336 - binary_crossentropy: 0.6432 - auc: 0.6800 - precision: 0.6290 - recall: 0.6446\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6431 - binary_accuracy: 0.6336 - binary_crossentropy: 0.6431 - auc: 0.6800 - precision: 0.6290 - recall: 0.6446\n",
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6431 - binary_accuracy: 0.6337 - binary_crossentropy: 0.6431 - auc: 0.6801 - precision: 0.6291 - recall: 0.6448\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6430 - binary_accuracy: 0.6337 - binary_crossentropy: 0.6430 - auc: 0.6802 - precision: 0.6291 - recall: 0.6449\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6429 - binary_accuracy: 0.6338 - binary_crossentropy: 0.6429 - auc: 0.6803 - precision: 0.6291 - recall: 0.6451\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 39s 782ms/step - loss: 0.6429 - binary_accuracy: 0.6338 - binary_crossentropy: 0.6429 - auc: 0.6803 - precision: 0.6291 - recall: 0.6451 - val_loss: 0.6274 - val_binary_accuracy: 0.6565 - val_binary_crossentropy: 0.6274 - val_auc: 0.7044 - val_precision: 0.6574 - val_recall: 0.6719\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 18/585\n",
      " 1/50 [..............................] - ETA: 10s - loss: 0.6420 - binary_accuracy: 0.6319 - binary_crossentropy: 0.6420 - auc: 0.6812 - precision: 0.6317 - recall: 0.6446\n",
      " 2/50 [>.............................] - ETA: 7s - loss: 0.6419 - binary_accuracy: 0.6345 - binary_crossentropy: 0.6419 - auc: 0.6816 - precision: 0.6306 - recall: 0.6494 \n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6419 - binary_accuracy: 0.6342 - binary_crossentropy: 0.6419 - auc: 0.6813 - precision: 0.6285 - recall: 0.6493\n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.6414 - binary_accuracy: 0.6345 - binary_crossentropy: 0.6414 - auc: 0.6821 - precision: 0.6294 - recall: 0.6511\n",
      " 5/50 [==>...........................] - ETA: 7s - loss: 0.6414 - binary_accuracy: 0.6348 - binary_crossentropy: 0.6414 - auc: 0.6823 - precision: 0.6289 - recall: 0.6512\n",
      " 6/50 [==>...........................] - ETA: 8s - loss: 0.6420 - binary_accuracy: 0.6339 - binary_crossentropy: 0.6420 - auc: 0.6815 - precision: 0.6281 - recall: 0.6502\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6418 - binary_accuracy: 0.6342 - binary_crossentropy: 0.6418 - auc: 0.6819 - precision: 0.6290 - recall: 0.6500\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6423 - binary_accuracy: 0.6337 - binary_crossentropy: 0.6423 - auc: 0.6811 - precision: 0.6284 - recall: 0.6485\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6423 - binary_accuracy: 0.6335 - binary_crossentropy: 0.6423 - auc: 0.6810 - precision: 0.6279 - recall: 0.6479\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6425 - binary_accuracy: 0.6332 - binary_crossentropy: 0.6425 - auc: 0.6807 - precision: 0.6277 - recall: 0.6468\n",
      "11/50 [=====>........................] - ETA: 7s - loss: 0.6424 - binary_accuracy: 0.6338 - binary_crossentropy: 0.6424 - auc: 0.6810 - precision: 0.6284 - recall: 0.6465\n",
      "12/50 [======>.......................] - ETA: 6s - loss: 0.6426 - binary_accuracy: 0.6338 - binary_crossentropy: 0.6426 - auc: 0.6808 - precision: 0.6287 - recall: 0.6463\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6424 - binary_accuracy: 0.6341 - binary_crossentropy: 0.6424 - auc: 0.6810 - precision: 0.6288 - recall: 0.6461\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6423 - binary_accuracy: 0.6344 - binary_crossentropy: 0.6423 - auc: 0.6813 - precision: 0.6293 - recall: 0.6463\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6423 - binary_accuracy: 0.6344 - binary_crossentropy: 0.6423 - auc: 0.6813 - precision: 0.6293 - recall: 0.6459\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6424 - binary_accuracy: 0.6343 - binary_crossentropy: 0.6424 - auc: 0.6812 - precision: 0.6291 - recall: 0.6456\n",
      "17/50 [=========>....................] - ETA: 5s - loss: 0.6424 - binary_accuracy: 0.6344 - binary_crossentropy: 0.6424 - auc: 0.6811 - precision: 0.6292 - recall: 0.6458\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6424 - binary_accuracy: 0.6345 - binary_crossentropy: 0.6424 - auc: 0.6812 - precision: 0.6289 - recall: 0.6461\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6422 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6422 - auc: 0.6814 - precision: 0.6291 - recall: 0.6460\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6422 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6422 - auc: 0.6815 - precision: 0.6289 - recall: 0.6463\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6421 - binary_accuracy: 0.6349 - binary_crossentropy: 0.6421 - auc: 0.6816 - precision: 0.6291 - recall: 0.6463\n",
      "22/50 [============>.................] - ETA: 4s - loss: 0.6421 - binary_accuracy: 0.6349 - binary_crossentropy: 0.6421 - auc: 0.6816 - precision: 0.6291 - recall: 0.6458\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6421 - binary_accuracy: 0.6350 - binary_crossentropy: 0.6421 - auc: 0.6816 - precision: 0.6294 - recall: 0.6453\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6420 - binary_accuracy: 0.6350 - binary_crossentropy: 0.6420 - auc: 0.6818 - precision: 0.6296 - recall: 0.6449\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6420 - binary_accuracy: 0.6350 - binary_crossentropy: 0.6420 - auc: 0.6818 - precision: 0.6299 - recall: 0.6444\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6420 - binary_accuracy: 0.6350 - binary_crossentropy: 0.6420 - auc: 0.6819 - precision: 0.6299 - recall: 0.6442\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6420 - binary_accuracy: 0.6350 - binary_crossentropy: 0.6420 - auc: 0.6819 - precision: 0.6299 - recall: 0.6441\n",
      "28/50 [===============>..............] - ETA: 3s - loss: 0.6421 - binary_accuracy: 0.6349 - binary_crossentropy: 0.6421 - auc: 0.6818 - precision: 0.6297 - recall: 0.6442\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6421 - binary_accuracy: 0.6349 - binary_crossentropy: 0.6421 - auc: 0.6817 - precision: 0.6297 - recall: 0.6443\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6421 - binary_accuracy: 0.6349 - binary_crossentropy: 0.6421 - auc: 0.6817 - precision: 0.6297 - recall: 0.6443\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6421 - binary_accuracy: 0.6348 - binary_crossentropy: 0.6421 - auc: 0.6817 - precision: 0.6296 - recall: 0.6443\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6421 - binary_accuracy: 0.6349 - binary_crossentropy: 0.6421 - auc: 0.6818 - precision: 0.6297 - recall: 0.6445\n",
      "33/50 [==================>...........] - ETA: 2s - loss: 0.6421 - binary_accuracy: 0.6348 - binary_crossentropy: 0.6421 - auc: 0.6817 - precision: 0.6296 - recall: 0.6445\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6422 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6422 - auc: 0.6815 - precision: 0.6295 - recall: 0.6444\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6422 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6422 - auc: 0.6815 - precision: 0.6294 - recall: 0.6446\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6422 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6422 - auc: 0.6816 - precision: 0.6294 - recall: 0.6448\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6423 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6423 - auc: 0.6814 - precision: 0.6292 - recall: 0.6448\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6423 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6423 - auc: 0.6813 - precision: 0.6293 - recall: 0.6449\n",
      "39/50 [======================>.......] - ETA: 1s - loss: 0.6424 - binary_accuracy: 0.6345 - binary_crossentropy: 0.6424 - auc: 0.6812 - precision: 0.6291 - recall: 0.6448\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6424 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6424 - auc: 0.6812 - precision: 0.6292 - recall: 0.6449\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6422 - binary_accuracy: 0.6348 - binary_crossentropy: 0.6422 - auc: 0.6815 - precision: 0.6295 - recall: 0.6452\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6423 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6423 - auc: 0.6815 - precision: 0.6294 - recall: 0.6451\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6422 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6422 - auc: 0.6815 - precision: 0.6294 - recall: 0.6451\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6422 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6422 - auc: 0.6815 - precision: 0.6293 - recall: 0.6449\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6422 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6422 - auc: 0.6815 - precision: 0.6293 - recall: 0.6448\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6423 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6423 - auc: 0.6815 - precision: 0.6293 - recall: 0.6447\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6422 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6422 - auc: 0.6815 - precision: 0.6293 - recall: 0.6446\n",
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6422 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6422 - auc: 0.6815 - precision: 0.6293 - recall: 0.6446\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6422 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6422 - auc: 0.6815 - precision: 0.6293 - recall: 0.6444\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6423 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6423 - auc: 0.6815 - precision: 0.6293 - recall: 0.6444\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - 40s 806ms/step - loss: 0.6423 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6423 - auc: 0.6815 - precision: 0.6293 - recall: 0.6444 - val_loss: 0.6273 - val_binary_accuracy: 0.6566 - val_binary_crossentropy: 0.6273 - val_auc: 0.7046 - val_precision: 0.6588 - val_recall: 0.6676\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 19/585\n",
      " 1/50 [..............................] - ETA: 10s - loss: 0.6404 - binary_accuracy: 0.6369 - binary_crossentropy: 0.6404 - auc: 0.6846 - precision: 0.6419 - recall: 0.6403\n",
      " 2/50 [>.............................] - ETA: 8s - loss: 0.6423 - binary_accuracy: 0.6334 - binary_crossentropy: 0.6423 - auc: 0.6814 - precision: 0.6330 - recall: 0.6392 \n",
      " 3/50 [>.............................] - ETA: 8s - loss: 0.6421 - binary_accuracy: 0.6338 - binary_crossentropy: 0.6421 - auc: 0.6814 - precision: 0.6323 - recall: 0.6407\n",
      " 4/50 [=>............................] - ETA: 8s - loss: 0.6427 - binary_accuracy: 0.6341 - binary_crossentropy: 0.6427 - auc: 0.6805 - precision: 0.6316 - recall: 0.6418\n",
      " 5/50 [==>...........................] - ETA: 8s - loss: 0.6414 - binary_accuracy: 0.6354 - binary_crossentropy: 0.6414 - auc: 0.6826 - precision: 0.6326 - recall: 0.6441\n",
      " 6/50 [==>...........................] - ETA: 8s - loss: 0.6417 - binary_accuracy: 0.6351 - binary_crossentropy: 0.6417 - auc: 0.6822 - precision: 0.6319 - recall: 0.6445\n",
      " 7/50 [===>..........................] - ETA: 7s - loss: 0.6420 - binary_accuracy: 0.6350 - binary_crossentropy: 0.6420 - auc: 0.6818 - precision: 0.6319 - recall: 0.6445\n",
      " 8/50 [===>..........................] - ETA: 7s - loss: 0.6421 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6421 - auc: 0.6815 - precision: 0.6316 - recall: 0.6448\n",
      " 9/50 [====>.........................] - ETA: 7s - loss: 0.6419 - binary_accuracy: 0.6351 - binary_crossentropy: 0.6419 - auc: 0.6818 - precision: 0.6315 - recall: 0.6461\n",
      "10/50 [=====>........................] - ETA: 7s - loss: 0.6421 - binary_accuracy: 0.6349 - binary_crossentropy: 0.6421 - auc: 0.6816 - precision: 0.6308 - recall: 0.6462\n",
      "11/50 [=====>........................] - ETA: 6s - loss: 0.6423 - binary_accuracy: 0.6350 - binary_crossentropy: 0.6423 - auc: 0.6813 - precision: 0.6306 - recall: 0.6465\n",
      "12/50 [======>.......................] - ETA: 6s - loss: 0.6421 - binary_accuracy: 0.6353 - binary_crossentropy: 0.6421 - auc: 0.6816 - precision: 0.6310 - recall: 0.6468\n",
      "13/50 [======>.......................] - ETA: 6s - loss: 0.6422 - binary_accuracy: 0.6352 - binary_crossentropy: 0.6422 - auc: 0.6815 - precision: 0.6308 - recall: 0.6468\n",
      "14/50 [=======>......................] - ETA: 6s - loss: 0.6423 - binary_accuracy: 0.6351 - binary_crossentropy: 0.6423 - auc: 0.6814 - precision: 0.6306 - recall: 0.6468\n",
      "15/50 [========>.....................] - ETA: 6s - loss: 0.6423 - binary_accuracy: 0.6350 - binary_crossentropy: 0.6423 - auc: 0.6813 - precision: 0.6303 - recall: 0.6467\n",
      "16/50 [========>.....................] - ETA: 6s - loss: 0.6423 - binary_accuracy: 0.6352 - binary_crossentropy: 0.6423 - auc: 0.6814 - precision: 0.6301 - recall: 0.6467\n",
      "17/50 [=========>....................] - ETA: 5s - loss: 0.6423 - binary_accuracy: 0.6350 - binary_crossentropy: 0.6423 - auc: 0.6813 - precision: 0.6300 - recall: 0.6460\n",
      "18/50 [=========>....................] - ETA: 5s - loss: 0.6424 - binary_accuracy: 0.6349 - binary_crossentropy: 0.6424 - auc: 0.6812 - precision: 0.6300 - recall: 0.6454\n",
      "19/50 [==========>...................] - ETA: 5s - loss: 0.6424 - binary_accuracy: 0.6348 - binary_crossentropy: 0.6424 - auc: 0.6811 - precision: 0.6298 - recall: 0.6449\n",
      "20/50 [===========>..................] - ETA: 5s - loss: 0.6425 - binary_accuracy: 0.6345 - binary_crossentropy: 0.6425 - auc: 0.6809 - precision: 0.6298 - recall: 0.6440\n",
      "21/50 [===========>..................] - ETA: 5s - loss: 0.6424 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6424 - auc: 0.6810 - precision: 0.6301 - recall: 0.6436\n",
      "22/50 [============>.................] - ETA: 5s - loss: 0.6424 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6424 - auc: 0.6810 - precision: 0.6302 - recall: 0.6433\n",
      "23/50 [============>.................] - ETA: 4s - loss: 0.6424 - binary_accuracy: 0.6345 - binary_crossentropy: 0.6424 - auc: 0.6810 - precision: 0.6300 - recall: 0.6432\n",
      "24/50 [=============>................] - ETA: 4s - loss: 0.6421 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6421 - auc: 0.6814 - precision: 0.6301 - recall: 0.6435\n",
      "25/50 [==============>...............] - ETA: 4s - loss: 0.6421 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6421 - auc: 0.6814 - precision: 0.6301 - recall: 0.6436\n",
      "26/50 [==============>...............] - ETA: 4s - loss: 0.6423 - binary_accuracy: 0.6345 - binary_crossentropy: 0.6423 - auc: 0.6812 - precision: 0.6299 - recall: 0.6433\n",
      "27/50 [===============>..............] - ETA: 4s - loss: 0.6422 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6422 - auc: 0.6813 - precision: 0.6299 - recall: 0.6435\n",
      "28/50 [===============>..............] - ETA: 3s - loss: 0.6422 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6422 - auc: 0.6813 - precision: 0.6298 - recall: 0.6437\n",
      "29/50 [================>.............] - ETA: 3s - loss: 0.6423 - binary_accuracy: 0.6345 - binary_crossentropy: 0.6423 - auc: 0.6812 - precision: 0.6297 - recall: 0.6435\n",
      "30/50 [=================>............] - ETA: 3s - loss: 0.6424 - binary_accuracy: 0.6344 - binary_crossentropy: 0.6424 - auc: 0.6810 - precision: 0.6295 - recall: 0.6434\n",
      "31/50 [=================>............] - ETA: 3s - loss: 0.6424 - binary_accuracy: 0.6345 - binary_crossentropy: 0.6424 - auc: 0.6810 - precision: 0.6295 - recall: 0.6433\n",
      "32/50 [==================>...........] - ETA: 3s - loss: 0.6423 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6423 - auc: 0.6812 - precision: 0.6296 - recall: 0.6434\n",
      "33/50 [==================>...........] - ETA: 3s - loss: 0.6424 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6424 - auc: 0.6810 - precision: 0.6294 - recall: 0.6435\n",
      "34/50 [===================>..........] - ETA: 2s - loss: 0.6425 - binary_accuracy: 0.6344 - binary_crossentropy: 0.6425 - auc: 0.6809 - precision: 0.6292 - recall: 0.6434\n",
      "35/50 [====================>.........] - ETA: 2s - loss: 0.6424 - binary_accuracy: 0.6345 - binary_crossentropy: 0.6424 - auc: 0.6810 - precision: 0.6292 - recall: 0.6436\n",
      "36/50 [====================>.........] - ETA: 2s - loss: 0.6424 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6424 - auc: 0.6811 - precision: 0.6294 - recall: 0.6436\n",
      "37/50 [=====================>........] - ETA: 2s - loss: 0.6423 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6423 - auc: 0.6812 - precision: 0.6296 - recall: 0.6436\n",
      "38/50 [=====================>........] - ETA: 2s - loss: 0.6422 - binary_accuracy: 0.6348 - binary_crossentropy: 0.6422 - auc: 0.6813 - precision: 0.6297 - recall: 0.6437\n",
      "39/50 [======================>.......] - ETA: 1s - loss: 0.6422 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6422 - auc: 0.6813 - precision: 0.6296 - recall: 0.6437\n",
      "40/50 [=======================>......] - ETA: 1s - loss: 0.6422 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6422 - auc: 0.6812 - precision: 0.6294 - recall: 0.6437\n",
      "41/50 [=======================>......] - ETA: 1s - loss: 0.6423 - binary_accuracy: 0.6347 - binary_crossentropy: 0.6423 - auc: 0.6812 - precision: 0.6294 - recall: 0.6438\n",
      "42/50 [========================>.....] - ETA: 1s - loss: 0.6424 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6424 - auc: 0.6810 - precision: 0.6293 - recall: 0.6439\n",
      "43/50 [========================>.....] - ETA: 1s - loss: 0.6423 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6423 - auc: 0.6811 - precision: 0.6293 - recall: 0.6440\n",
      "44/50 [=========================>....] - ETA: 1s - loss: 0.6422 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6422 - auc: 0.6812 - precision: 0.6294 - recall: 0.6441\n",
      "45/50 [==========================>...] - ETA: 0s - loss: 0.6422 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6422 - auc: 0.6812 - precision: 0.6294 - recall: 0.6442\n",
      "46/50 [==========================>...] - ETA: 0s - loss: 0.6423 - binary_accuracy: 0.6345 - binary_crossentropy: 0.6423 - auc: 0.6811 - precision: 0.6294 - recall: 0.6442\n",
      "47/50 [===========================>..] - ETA: 0s - loss: 0.6422 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6422 - auc: 0.6812 - precision: 0.6294 - recall: 0.6443\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48/50 [===========================>..] - ETA: 0s - loss: 0.6422 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6422 - auc: 0.6812 - precision: 0.6295 - recall: 0.6444\n",
      "49/50 [============================>.] - ETA: 0s - loss: 0.6422 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6422 - auc: 0.6813 - precision: 0.6294 - recall: 0.6445\n",
      "50/50 [==============================] - ETA: 0s - loss: 0.6421 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6421 - auc: 0.6813 - precision: 0.6294 - recall: 0.6445\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(Worker pid=226680, ip=172.168.0.103)\u001b[0m Epoch 19: early stopping\n",
      "\u001b[2m\u001b[36m(Worker pid=259340, ip=172.168.0.107)\u001b[0m Epoch 19: early stopping\n",
      "\u001b[2m\u001b[36m(Worker pid=314019, ip=172.168.0.104)\u001b[0m Epoch 19: early stopping\n",
      "\u001b[2m\u001b[36m(Worker pid=310081, ip=172.168.0.105)\u001b[0m Epoch 19: early stopping\n",
      "\u001b[2m\u001b[36m(Worker pid=279949, ip=172.168.0.102)\u001b[0m Epoch 19: early stopping\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m \b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r\n",
      "50/50 [==============================] - 39s 796ms/step - loss: 0.6421 - binary_accuracy: 0.6346 - binary_crossentropy: 0.6421 - auc: 0.6813 - precision: 0.6294 - recall: 0.6445 - val_loss: 0.6272 - val_binary_accuracy: 0.6568 - val_binary_crossentropy: 0.6272 - val_auc: 0.7044 - val_precision: 0.6566 - val_recall: 0.6757\n",
      "\u001b[2m\u001b[36m(Worker pid=314929, ip=172.168.0.106)\u001b[0m Epoch 19: early stopping\n",
      "Training time is:  1185.4728910923004\n"
     ]
    }
   ],
   "source": [
    "def label_cols(column_info):\n",
    "    return [column_info[\"label\"]]\n",
    "\n",
    "def feature_cols(column_info):\n",
    "    return column_info[\"wide_base_cols\"] + column_info[\"wide_cross_cols\"] +\\\n",
    "                  column_info[\"indicator_cols\"] + column_info[\"embed_cols\"] + column_info[\"continuous_cols\"]\n",
    "\n",
    "start = time()\n",
    "estimator.fit(data=train_tbl.df,\n",
    "              epochs=epochs,\n",
    "              batch_size=batch_size,\n",
    "              steps_per_epoch=steps_per_epoch,\n",
    "              validation_data=valid_tbl.df,\n",
    "              validation_steps=val_steps,\n",
    "              callbacks=callbacks,\n",
    "              feature_cols=feature_cols(column_info),\n",
    "              label_cols=label_cols(column_info))\n",
    "end = time()\n",
    "print(\"Training time is: \", end - start)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f8470e5",
   "metadata": {},
   "source": [
    "After 19 rounds of training, we get validation [auc](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) 0.7044, [precision](https://en.wikipedia.org/wiki/Precision_and_recall) 0.6566 and [recall](https://en.wikipedia.org/wiki/Precision_and_recall) 0.6757. The total training time is around half an hour.\n",
    "\n",
    "After the training, we save the trained model to be used for serving."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1f21b787",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-03-23 11:26:01.562497: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2022-03-23 11:26:01.562564: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2022-03-23 11:26:01.562601: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (aep-001): /proc/driver/nvidia/version does not exist\n",
      "2022-03-23 11:26:01.563329: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-03-23 11:26:02.425917: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: recsys_wnd/assets\n",
      "Stopping orca context\n"
     ]
    }
   ],
   "source": [
    "model = estimator.get_model()\n",
    "tf.saved_model.save(model, \"recsys_wnd/\")\n",
    "\n",
    "stop_orca_context()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.13 ('friesian')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "6c9c6c36520220f3300aaef54e998202ae22aa0de07e5d4c5d990becb2915fb2"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
